Introduction To Titanic Kaggle Competition | Part 1

Key Takeaways

hello all right my name is uh fuk dang I'm the senior data engineer of data science dojo and I'm here to walk you through day two's homework I hope you've been enjoying uh so far of the boot camp okay so quickly just go to portal. science dojo.com all the homework is laid out there for you um if you prefer for a video this is a video and walking you through the homework but just so you know the homework is elaborated in on both sections so there are two parts of the homework the first part of the homework is to apply what you learned today so basically take uh the Titanic data set and apply a predictive model to it so go ahead and use our part or if you've gotten to random Force go ahead and use a random force model okay that's the first part of the homework the second part of the homework is to actually enter into a kagle competition okay so both the homeworks are elaborated here um I'm just going to go talk and show you how to do the kago competition real quick okay so this really is your data science Capstone project for this course so by the end of Friday basically you'll be working from Tuesday all the way to Friday to perfect your model and then you'll be ranked among your peers your peers being basically everyone at the boot camp and then there are prizes on the line so I'll talk more about the prizes later but for now you'll this whole page talks you through how to create a kagle account how to submit and how to do all that good stuff now I will talk you through that also here as well okay so um what you want to do is you want to Google kaggle Titanic okay so notice that we've actually entered you into a kaggle competition since day one so the Titanic data set actually comes from this kaggle competition and what is kaggle well kaggle is a crowdsourced way of doing data science so real companies like Home Depot Liberty Mutual All State Netflix they come together and post real data sets and from these real data sets there is a data mining problem and you're ranked among your peers as you do these data mining problems on what are called leaderboards okay so the Titanic competition is basically the introductory uh kagle competition homework uh that that we'll do uh together okay and then if you notice there's a if you go to data okay in this thing there are a bunch of data sets that are associated with this uh kagle competition so if you notice here there is a train. tsv and let me tell you what that is real quick so you notice that throughout this cackle competition you've been given this data set with 189 or 191 rows right this is this is the training set okay this is the set that you've been working with although some of you should have been kind of suspicious if you've been paying attention to history the Titanic boat actually housed about 2,000 people yet we only have 891 passengers I wonder where the rest of the other passengers went well it turns out kagle actually is withholding the other passengers in this test set okay so your homework is actually to basically build a predictive model your Capstone uh is to build a predictive model on this training set and to apply it to this test set so I'm going to go ahead and download this test set we can see what is inside of it okay so if you open up this test set you will notice that the passenger ID starts at 892 okay so these are the remaining passengers that were on the Titanic but you also notice that we have one less column you notice that survived is now missing okay that is your job okay you're supposed to predict whether or not these people will survive or die okay so notice that that is all the kagle competition is it's kago wants your answers they want to know whether or not individual passengers lived or died for example passenger 897 did they live or die based upon these demographical conditions that are that are going to be read in by your Ive model okay so I'm going to show you real quick how to submit to kagle um for the purposes of just uh uh introduction so uh for tonight's homework you don't need to hook up a predictive model and submit to kagle you just need to just submit okay and I'm going to show you how to submit so kago wants two things from you it wants passenger ID right and it wants basically survive did the person that is corresponding with that passenger live or die so kago just wants two columns from you so the fact that these columns are here ah irrelevant so we're going to delete it so kago wants uh a column called passenger ID and notice that the I is capitalized and the p is capitalize and it's one word okay and it also wants a column called survived notice that it's past tense and there's a capital S kago will check for that and we're going to build a very simple model a model where everyone dies okay so you notice that if everyone dies uh then that this is going to be a very uh basically it's not even a predictive model we're just going we're just going to say if you step on this boat you will die but notice that if you remember from day one when we did data exploration when we looked at the class distribution of survived versus dead we we noticed that there was about a 62 is% chance of death just by stepping on the boat so actually by saying everyone died we have a statistical likelihood of saying of be doing better than a coin flip right doing better than 50% so I'm going to go ahead and say everyone dies here and I'm going to save that as a CSV so I'm going to go and save this as a my own model so everyone dies. CSV and I'm going to save that all right and what you need to do is you need to go to kago and upload this file so go ahead and make a submission there's a make submission button here so click on make submission and then go ahead and we'll upload a submission in here so everyone dies. CSV and we'll go ahead and submit that all right so it just so happens notice that we notice that we don't even give kaggle uh our predictive model we just give kagle the answers that makes it so we can build a predictive model in Python asure it doesn't matter it is now class agnostic they only care about your answers and notice that we are just submitting predictions to kagle and kagle is actually going to score this and kagle is actually going to be able to give you an accuracy out of this that's because they actually hold the true labels kagle actually knows whether or not the person lived or died and if you remember from evaluation if you compare predicted versus actual you'll get a confusion Matrix right so you submit predictions kaggle has the actual from that kaggle BS to confusion Matrix from the confusion Matrix you get accuracy and notice that it spits me out and accuracy and it says my submission got a 62% accuracy and I rank 5,517 in the world okay all right so the Capstone here is basically we're going to enter all of you guys into a kagle competition within the class okay and to enter yourself into this kagle competition save the name that appears on the kagle leaderboard so notice that I'm fuk Hong so I'll save my username as fuk H and then I'll go ahead and go back to that kagle submission homework and I'll paste it into this form down here so this form down here will actually uh go ahead and enter your KAG username into our internal leaderboard okay and on Friday after lunch uh we're going to end the kagle competition wherein the first prize the first place winner basically the first the person that ranks highest will get a prize right the prize will be an advanced statistical uh r book okay and it's a very good book if you want to do some of these these extra Advanced Data Mining processes in R that's in there and notice that we only can teach you so much that book actually contains a lot of the other stuff that we couldn't teach you for example there is actually more than one way to cross validate right we taught you just kfold cross validation but there's also leave one out cross fold validation right so there's four other ways to cross validate that we were not able to cover in class and that book covers that and then the second and third place winner will get an oal book called doing data science uh I also really enjoy that book I would I was raised by o Al and hopefully you will be as well okay and more importantly yes I know you can buy these books I know you can go ahead and just you know kind of pass this off but this is really important you want to do this kago competition and be able to ask the instructor questions while you're still in class right cuz there's actually lot of minute little steps to go along the way here uh that might that might basically you when you go back to work and you try to work on your own Calo competition or your own data sets Okay but more importantly your honor is on the line you have to defend your honor and basically and you will get big bragging rights from all this okay all right now happy modeling