How do I use pandas with scikit-learn to create Kaggle submissions?

Key Takeaways

hello and welcome back to my QA video series about the pandas library in Python and the question for today comes from an email i received from rai ed and he says can you please do a short video on creating pandas dataframe objects from numpy arrays and then writing to a CSV file I had problems submitting the Titanic problem on Kaggle but later found some code that I copied to get the job done okay excellent question so let's start at the beginning here what is Kegel so Kaggle is a a popular platform for doing competitive machine learning ok and what is machine learning in a sentence I would say that machine learning is the semi automated extraction of knowledge from data okay now I'm not going to focus in this video on talking about machine learning I have a video called what is machine learning and how does it work I'll link to it in the description below if you're interested in learning more ok so in this video I am going to focus on how to use pandas in partnership with scikit-learn for machine learning ok so as always we need an example data set and I'm going to use the data set from kegels Titanic competition you can follow along and by the end of this video you will have a submission file for cackled ok so we're going to import pandas as PD and then I'm going to create a data frame called train and it'll be PD read CSV and I'm going to pass it the URL fit dot Lee slash Kaggle train ok and it's called train because this is our training data it's the data that our model is going to learn from ok so let's look at the head and what we have is each row represents a passenger aboard the Titanic and the Titanic was a ship that sank a long time ago so each passenger it has some attributes of that passenger as well as this column survived which is one if the person survived and zero if the person did not so the goal of this competition is off is for the tests at which we'll get to later for the test set predict survival based upon other characteristics of the passenger okay so first thing we need to do is create our feature matrix X these are the features the columns that our model is going to learn from okay so I'm going to actually just create a list a Python list called feature calls and it's going to be a list of strings and I've selected P class and parch which stands for parents and children number of parents children P classes for passenger class I'm just creating a Python list and this is these are the two columns I've selected as my features okay so I need to create X my feature matrix and we're just going to say train dot lope which is how I select rows and columns from the data frame so I want all rows and I want the feature calls columns okay so this is a data frame a panda's data frame I've created and we can check the shape and it is 891 rows by two columns okay so there's my features now I want to create my response factor also known as the target factor the thing you are trying to predict and that's the survived series so I'm just going to say y equals Train survive ok now this as I said is a panda series and we'll check the shape and it's 891 by nothing it's it's got only got one dimension okay so x and y are pandas objects there's no need to actually convert them to numpy arrays um and in fact scikit-learn will understand the x and y objects as long as they're the right shapes and they are fully numeric okay so no need to convert them to different object types alright now we're ready to build our scikit-learn model and i'm not going to explain this code because it would take quite a while I actually have a video series which I'll link below um called introduction to machine learning with scikit-learn it's 4 hours long and I would love for you to check it out if you're interested in learning about machine learning okay so here's my quick code to create a classification model so I'm going to say from SK learn dot linear linear model import logistic regression which is a classification model then I'm going to instantiate it let's just stick regression ok and then I'm going to fit the model to my training data log reg dot fit X Y ok so we have now fitted our machine learning model and I'm going to return to some pandas code so we need to read in our test data ok the test data is the data we're going to make predictions on and so I'm going to say test equals PD read CSV bit dot leak slash kaggle test okay and if we check out the head we will see that it looks very similar except it is missing the survived column and the reason for that is because we are predicting that we don't know the survived column for these people that is what we are predicting okay so I need to create a new X so I'm going to call it X new from the testing data okay so I'll say test dot local rows and these two feature columns okay and we'll check the shape woops x new dot shape and it is 418 rows by two columns so there's 418 observations in the testing data we need to make 418 predictions okay so we're ready to make our predictions one more line of scikit-learn code and I'm going to say new pred class meaning for this new data what are the predicted classes it's a classification problem and I'll just say log reg predict X nu ok now I've actually got everything I need I just need to put it in a CSV file kaggle asks for a CSV file with two columns the first column is going to be the passenger ID from the testing set so test dot passenger ID and it's just these numbers it's a panda series okay and then the second thing it wants is the predicted classes so new pred class and it's just a bunch of zeros and ones our 418 predictions okay so how do I create a CSV file with these two columns I'm actually just going to use the data frame constructor so PD data and there's lots of different types of objects you can pass to a data frame and it will figure out how to create the data frame you could pass it an umpire array you could pass it a list of lists I will cover that in a future video but for today I'm going to pass it a dictionary okay and I'm going to say a passenger ID : test dot passenger ID okay and then I'm going to say survived : new pred class okay so what am i doing I'm saying I want two columns that should be called passenger ID and survived and then I'm going to put this will be the test that passenger ID are the values that I'll put in that series new pred class are the values that I'll put in this series and pandas will automatically align them next to one another okay so we create that and here's what it looks like now one thing I need to make sure is that passenger ID is the first column now dictionaries are unordered so it's impossible to guarantee that passenger ID will come first so actually the easiest way to ensure a particular column is the first one in the data frame is to set it as the index so I'm going to just say dot set index passenger ID okay so I've accomplished so all I was trying to do is make sure it is the very first column there are other ways to do it I just like this way okay final step is we're just going to use a data frame method called to underscore CSV and all you have to do with to underscore CSV is to just pass it the name of a file that you want to create so I'm going to just say sub dot CSV which is short for submission a Kaggle submission okay so when I run that line of code it creates that CSV in my working directory it automatically includes the index of the data frame as the first column if you ever want to exclude the index that's just a parameter to to CSV that you can use okay so if you've been following along you could submit this file to Kaggle right now it's not going to perform very well but my goal in this video is to show you the workflow for using um pandas and scikit-learn together okay so as always I'm going to end with a bonus and here's the bonus um have you ever wanted to save a Python object such as a data frame to disk okay so um there's one way you can do that just in python for any Python object but with data frames um we are going to use something a method called to pickle because the object are called pickle objects okay so we'll just say train dot to underscore pickle and all you have to pass it is the name of the file you want to create and I'll just say train dot PKL so I'm just taking the train data frame and saving it to disk I could put it on a flash drive bring it to a different computer or something like that and load it right up without you know going through all the steps I had to do to create the data frame so I'll pickle it that's what it's called and then when you want to read it into pandas we're just going to say P dot read pickle okay and again we just pass it that same file name and it will look for that file in the working directory and read it in and we've now got our data frame back okay so that is it for today thank you so much for joining me as always please click Subscribe if you'd like to see more videos like this please leave me a question or a comment in the comment section below I'd love to hear what you have to say and again that's it so thank you so much for joining me and I hope to see you again soon