Feature Engineering and Predictive Modeling | Data Analytics with R and Azure ML | Community Webinar

Key Takeaways

we are going to explore features and visualize some teachers and eventually will and I will give a quick overview to a new tool that came out it's called machine learning so the goal today is introduced for those of you who have not played around with I show some graphics given Yamaha and not just the graphics capabilities we are also going to show how do you really analyze and slice and dice data and try to understand what what should be the thinking behind when you are trying to build a predictive model and so initially I will be doing only the best graphics the graph the package that ships with this are but eventually I will show some of the capabilities of GE not to and give some examples there so as I said I think the goal of the talk is to give an idea of the higher level thinking process that explains how to visualize the data how to understand your future features I think based on my own experience what happens is that most people once they have the data set and they know certain algorithm what they will do is just able to take the greater setting just give it to the algorithm without really doing their due diligence what what can the algorithm really do because all the data need to go in there without word about not so so those also other go through that thinking process and we'll see how usefully so this is the agent of today we're going to start with some sample data sets most some of you may be familiar this iris and Motocross data set we will go to titanic data sets and when we get there I will explain what it is looks like and then we will do some visualization to teach Atlantic and if if we have time I will try to build a predictive model right in front of the Vice I have such our code and then the outputs and everything and then we'll see it looks like and so the slide deck is going to be completely reproducible if you go and take the slides and start talking and pasting code from the slide it should work so I have I have the wave equation the searchlight decades that I would write something take the output paste the code so I think that should should be working and then I will show this machine learning show you again it's it's very useful because the great thing about machine learning as a service is that you don't have to worry about not everybody has a machine with 32 gig of 64 gig of computer and then once you have even if you have one of those machines what if you want to do another experiment what if you want to do twice on recreational preparation with widget aggression and lasso at the same time how are you going to do it because we have only one machine but when you have a service out there you can create as many experiments and then the service will actually take here take care of giving you the machines so and then it's going to be a very brief and short tutorial because we have a lot of slides and let's see how it goes suppose we will go to the virtualization aspect of things so usually this session is donation ductus it's conducted and very interactively I will show the code and then everybody will copy and then we'll type in the IDE but given we have only one and a half hour I think this is about three hours worth of work if I do it in productively but I have to really skim through it but it shouldn't really impact your and this guy if you if you go back and if you look at it it will make sense and when you go back and if you review it I think it should you should be able to connect the dots okay so so there's a saying their data beats algorithm and which I tend to agree right so if you have more data even you a simple algorithm should work but there are some caveats there if you have depending upon how much data you have right so if actually the returns from data they are going to diminish and also data quality and variety matters so if you if you're training a search algorithm a search ranking algorithm and the data that's going in it has a lot a lot of pause it so it has a lot of automated traffic and if you pre your ranking algorithm use based on those bots and they can be scraping your website your search engine and they're not even clicking on anything they're just creating or sometimes it's some SEO company in that they are clicking and going to next page in next page the next page you see there is their client landing on the ranking right so so those are not typical those are not the typical behaviors and you don't want to optimize your search engine for those behaviors so how do you know how to train the other right so you have to really worry about the quality of the data and data quality is a big issue wherever there is data you have data quality is the biggest the bigger social equity are and of course variety right so if you are building a model that should work for multiple markets and you have only data for US market does it make sense it may or may not work but definitely want to have a business all the all the possible varieties in your data set and you a decent performing learning algorithm is still needed you cannot just just have a very dumb algorithm with with no due diligence so it still needed and of course most importantly you should be extracting the useful features of the data if you stick everything into into the algorithm sure the algorithm will do a decent job but your performance algorithm performance is going to be great right so perhaps only ten features are important you are giving the algorithm 1000 features eventually many other ones you will figure out which ones are important but it's not the right way to do things okay so just we are just jumping right into right so so usually when I do a visualization the first thing that I do is I would look at the theater said I would eyeball the dataset maybe first couple of rows maybe one first thousand rows oh look at the dataset and we understand what is it that the data has so this is the data set I brought it to the copy in case the first few rows here so what this is doing is let's do the ships with base our package if you install our I display the second is going to be there and if you want to get more details on what this data set is you can go to so what this data set is the it has certain few columns that describe each of the different parameters about a different species of novel so first parameters is probably sepal length and petal length and sepal width and petal width so this this is the four columns which are the predictors and the last column is is the class and when I load this data so the first thing is data is it just loads that data into memory the second command hit of iris it actually gives you first few rows and you will see that I think I didn't basic here but in later example I will use that and the third thing okay and the tractor Mack is usually when you have some numeric predictors what you would want to do is you would want to see really what are the what are the ranges of values so if I give you a data set that has some ages and bits of some male and female patients rights so what you would want to do is you would want to put up create a box plot and look at both categories so you can create a box plot that has the whole population both male and female but you you should really look at the heights and weights segmented by the gender because gender is a it's a useful segment here right so in this case you are looking at three species so species in the class and we have species virginica and it shows you how the values are there are varying here so the first is you can you can call it the minimum value all right so this is the minimum value the top line is the maximum value I don't want to be precise here I think it's it's 1.25 of the interquartile range on the bottom and the top I don't want it there let's think of this as the minimum and the maximum and the dark line in the bed middle it shows the median so this is the most common or right in the middle value and so you have over the first two species and seconds each infant species and I was able to get this visualization just with this simple command I say I what I said was I want to plot the sepal length I want to box plot of several lengths for all species and data is iris I'm telling if the data is iris and then they actually give and then the text label is vici then my level is several lengths thanks Siri simple so what what insights if any can be draw from this box clock any any thoughts here so let me ask you this so if if you what would be the two two times that will be difficult to distinguish if you have to distinguish between them based on only the several times right the last two right because if you see this sitios I have the so this is the minimum the twenty-fifth percentile the 50th percentile it's empty that person right right so most of the values in sedusa really lie under five point five right so really if if you have any any value that says it's most likely not see Joseph right but if there is a value that six of six point five then you can actually confuse between the the next two because there where the length values are pretty close to each other and this will become more clear I think if you're looking at it for the first time I think it will take some kind of training or adaptation but really what's happening is let's see what if you're trying to distinguish between these two any of these classes it's excluded bit difficult to the distinguish between the last two classes versus if you have to distinguish between the first class of the last class because if you look at the sedusa the maximum value is around six and virginica the minimum value on check so I think you can really define a babe really dumb classifier ruthless classified if length is less than and if you know that this these are the only two classes when you need assist you can really just say it length is 6 less than 6.0 it's a cosa otherwise it's going to be a good machine learning algorithm in this case it's just a basic who would be a chopper okay and we can start getting fancy if I say not equals true it draws these not not not appear and rest of the arguments are the same and what that means is that it's it's given you believe Mr 95% confidence around median and it gives you this confidence that most of the time the median is going to lie on this boundary so it gives you some additional insight into what what is the most likely value of a median and this is just something in not equals true I just added a visualization purposes okay okay so how do you save plots you can see it lots and many different formats list with them in the table on the left and really the way you save a plot is you start with the first line PDF if you want to store that as a PDF if it's a nd you do it in the PHA Beckham whatever and then you draw the draw that plot that you're trying to draw and then just like you would close the programming handle after you read the screen you would say DEFCON half and then and believe me it will be in whatever directly you are in you can just experiment to definitely okay so so we can we can look at the numeric values and in a boxplot constant right and what boxplot does is it gives you a range of the values it gives you it gives you an idea of what the value what the different values are but we can also have density plots I think histograms is so you are getting some pockets there and if you look at the syntax here what I'm saying is so this is another data data section were two stars data set and what I'm doing is I'm locking the miles per gallon so this is this so this is a bunch of cards that have different mileages and you have ranging from ten bytes per gallon and you can you can see most of the guards are around 15 months again I think this is an older data set from good ol 90s and 80s I guess because I don't think of all the other mileages of the credit cards and eternity in the market I think it should have shifted towards 35-ish right so now we have smaller cars and more but what this shows us I mean within a single command again right so I just use a single command I said great sequence again I split it into ten distinct buckets color that's the color miles per gallon and it shows me an ecosystem that means that I'm plotting a histogram what this is doing is it's it's plotting the mileages and their frequencies on so what what is the most common mileage from a car in more intensity fashion right so it's more for probability density function and the same thing so this is missing okay so there's a okay so so this this thing should be very very complex syntax I'm not going to go into the syntax what this is doing is it loitering it's comparing multiple multiple with density plots and this is what it looks like right so what what this data I did was it flooded the mileages of a four cylinder in a six cylinder and an eight cylinder so we have these start with different mileages and then this is what it looks like so any insights that you get it should do this I think I should have included so this is what the design looks like right so this is what the okay so what this is is it it has a bunch of different types of cars and the first problem is the mileage per gallon which is most likely I think this is a regression problem you have to predict the mileage if you if I put it that way so first first column is my little polka gallon the second column is cylinder number of cylinders the third one is probably displacement I guess and then horsepower I don't know what here at it is weight of the car and then so on there is a bunch of different things so let's actually perform some hypothesis here what do you guys think what do you guys really think is going to impact the mileage per gallon but what is me to matter if you have to the car okay so that's one candidate okay what else number of cylinders right what else displacement okay so big so if the three candidates are bait displacement and number of cylinders and we will see whether our hypotheses are true or not so does anybody yeah so yeah so definitely that will impact so but in this case let's take the big and the v8 a number of cylinders and displacement as some of the candidates and see which one has the most impact okay okay any insights it does number of cylinders have any impact based on the distribution here so what this what this is the blue density plot tells me that for eight cylinder cars the 15 miles per gallon is the most likely the mileage that you will get out of it it's another car then you have a six cylinder car and twenty miles per gallon is the most most common mileage that you would get I'm trying to put it in layman's terms right I'm assuming that not everybody knows and then four four four cylinder car I think around 23 ish there is a peak and there is another peak at 32 right so do you think if this finding is consistent with what we just said or what we just discussed okay right so we have we can clearly see over here that it's roughly consistent with what but we definitely see that there are four cylinder cars that are not so efficient they are on this side and and there are I think some cars that are likely to be its Olympic routes and they are probably on the more efficient side right so if you just use the number of cylinders which two classes are going to be most difficult to classify I mean which classes will be so if you if you if you're trying to decide whether it's a force on a guitar or six-cylinder car or an 8-cylinder car just based on miles per gallon which two are going to be motion to the six and eight point now six and eight is Kranthi was confusing I think six is the problem here hi today if you so if you try to differentiate between six and four you're still going to be a problem and six and eight are going to be a problem again right because six is sitting right in the middle but if you have to do for in six four and eight chances are that not many cars with had later than then we still a mileage of eight cylinder right so it's basically it's common sense that we already know based on a VIP driving cars right so we know that but it's good to see all that we know in right in front and then just to see if it makes sense to us okay and this is this is cool right and if you go back this is a beef index right I mean this is a lot of realize and I will so I could have to start it physically widget lot to but you wouldn't have appreciated how how nice easy plotting is if you hadn't seen all of this right so I just wanted to go through this before we get to GT traffic so discuss ggplot very briefly and then you're not going to do grammar structure and things and thicknesses of video because this is not intact yet the idea is to just be able to explore each other and understand each other it may make sense to us okay so this is your exercise set course for home and I will still show the outputs you still have the commands but really IX I will just quickly skim through these and other I want all of you to actually go back and then just try to run these commands and see if you can if they give you any insights because I will be showing many of these things again gg-got tools I don't want to do this so this is the scatter plot so this is Garvey on x-axis so this is how we do it right attach actually loads data set in a memory in a way that you can refer to the columns and so it has attached it's not mine but really if you look at this I'm using the command plot and I'm plotting date against the mileage and then main if again it's used all over our so you should get used it but not for given the name label X label and by label and that says PCH is the type of chart that I chose for plotting and DCA to put 16 is something else in 21 is something else so you can so how do you create this plot and can anybody give me an idea as the car weight increases the mileage per gallon is getting lower how about this what's the difference between this and this so forget about the lines and the straight line and this curve the plastic triangle and surface basically I overlaid for this we didn't have any disability to the number are the number of cylinders right but a lot of times you would want to know the segment's in your data think about this so you ship a product in production and people are using that in things look okay but suddenly you realize you look at the segment's you have a chrome segment a safari segment and ie segment or some other miscellaneous browsers it means it is entirely possible that the behavior in one of the browsers is different right I'm just giving an example because so segmentation is extremely important when you doing data analysis you want don't really don't really accept whatever you have right so if you if you think you have some some numbers that you have try to slice and dice them into data sets if it's population tried by into a different young and old or gender deer segments or geography based segments so try to use demographics or income so there can be so many things and it's the same idea right so over here we didn't have any visibility into the different number of cylinders and what is a valuable insight that we have here is that it turns out that this should be are some there are some six-letter cars that have this is eight right so there are some six-cylinder cars that have inefficiency as bad as as infinity traffic so this is actually giving us more insight it's not just any random points we know what category each point belongs to so I will discuss this again and the Xbox but what this is really doing is sometimes you want to do sometimes you would really want to see a big bird's-eye view an overview of how each variable is changing against so in this case any so this is must be sergeant now and this this project should be confusing the first time and people but if you look at this here right so what this means is that if I look at this plot this is for the H versus T right so go to the intersection here and really so this is me and this is T so any any idea if you fade is increasing what's happening to this variable called tierod increasing or decreasing what kind of correlation is this it's decreasing right so we can clearly see it that there is a pattern here that this is going down as the rate increases right but what about is placed in here can you see that so this is way and that accesses displacement what's the relationship there and this is the opposite right so then rate is increasing the displacement is increasing and I don't know anything about cars I mean I really I don't know I mean if this should be the case or not but the teacher suggests that this is the case if I do all the way out now this one is weight and then this one is a mileage per gallon right so we all have seen this that as the weight is increasing the mileage per gallon is decreasing and so you will have to probably start look at it a couple of times before you can actually absorb it okay so if I look at this plot does it give me any information about how many cylinders the car has no right and it can be valuable information perhaps overall it seems like the train but if you consider individual segments the train may be reverse and it's not the case in this data set but it's entirely possible let's look at this again this is a package called lattices it's another popular package and I'm using this here right okay so this is for the iris data and let me be sure you guys are news data so it starts making sense okay so this is what I'll state as it looks like so you have the first predictor separate line the second predict a little bit the third predictor is petal length and petal bit after that and then last column is the class the category of flowers and there is only three okay I'm sure I can actually yeah so this is just a head right so I can know if you go and look at it it's okay I think it's not a big big data set so I can properly okay so I think that means equal supply okay so so this is the case now if you if you look at let's take for example this one so anyone accused to tell me which what kind of correlation this is describing here so we will go all the way here which is the sepal length and we go all the way up and petal length right so this is telling me it's official I am visualizing the correlation between the second line so if I had looked as assumed I mean just try to think in that way that you there is no color correct in that case this whole blob seems to be going in this direction right but when you go to the segment's does it does it give me any information here and so what is the information that I get right there is a different kind of correlation here versus this block right and I wouldn't have known if I didn't segment the data I would never have known this and that's why this is important any other interesting things I think if we look at except a little bit and Metrobank this one right here if I didn't look at the colors here what would be my conclusion no correlation right it's it's the blob is spread all over right is the x-axis changes Y is spread all over right again if I separate the dosa and versicolor and virginica I see a strong correlation here like this if this is if stubble but it's a petal length is increasing the sepal length is also increasing right so so you can see that as soon as ice segment lady dies as soon as I slice and dice the data the the correlation and any other insights then become much more obvious yeah and again so I can if you guys know what I can actually copy and paste this code but this is not going to be very clear okay so this this is some other examples you can have scatter plots with the trend lines and regression lines in it so this is a very rich plot in the sense that you can in the same plot you looking at the densities along the main diagonal how how the distribution is changing for mileage per gallon and you have the regression lines and the fix there you have each number of cylinders they are separated so it's a bit rich plot and again I think we cannot really spend time on that but this is this is giving you a lot of insight so we are doing all of this to get to the point when we are able to actually analyze the the Algona structure you get to Titanic later setter that's why I'm rushing to this so I will slow down a peg man we get to Titanic so it it will make more sense to you okay and this is to me scatter plots again the code is there you can see X I don't see any use for that like it's you can and then this is another one okay so we are there finally okay so now so we will slow down here a bit so this dataset is available I think you can get that on take up and so this data where I terminated from it was a CSV file and it was in this this location on I did cause I said Titanic towards being dot CSV and I was able to do anything note that data into something that's called a data frame so you can you can I think the closest example for this is it's a just like a database table so you can refer things by column by rows and so on and then if your data is tab delimited you do retreat on table and you can also specify if it's a limited by some other okay so can everybody you read that from but because now we are talking about really what we what we wanted to do we wanted to analyze this data set so people at the back end you see this so we have this wait a second so now let's now that we are at the problem that we wanted to solve let's try to build some hypotheses e're right so based on we don't know who survived so the problem statement here is we want to know who survived so we we have a sir we have some information about all the all the passengers that were on board and some survived and some didn't right so first of all we look at the data will I ball it and then see what is it that we have available if you have the passenger ID we have they were they survived or not zero means they didn't and one means they did survival and then we have the speed loss and this is peak losses the passenger class what was their class in temperature and then their name and their gender the age sibling in spouses on whether they had any siblings spouses on board or not and how many birthdays so and I think this is up to this variable has of the values of two eight seven so there were big family and then parents and children and then the ticket number the fear the cabinet aversion and this is the port of embarkation and I think one person's Berg and there was some other Southampton and third one I don't remember so this is the code for all of them okay so I want everybody to actually be more proactive yeah what what would be some of the hypotheses who is going to survive here okay so first hypothesis is the people and the last one they would survive and appear assuming that class 1 means the first class right service okay but that so usually I think this is true in this case right but I would caution against that because when we go to the data usually what happens is that the pay person is generating the data is not the person who's analyzing the data so don't assume things okay because the person may think that okay this is the class one in the sense yeah not in the better sense right in the order I mean maybe this is for the first class that I thought of that right so so it could be first class but different meaning so I think it's really obvious in this case but this these kind of conclusions happen when you the person generating the data and person consuming that it does not the same okay so class what else been in sure location of the cabmen and and we don't know that right so with the cabin if we put it that way the cabin would have it at any time there right so cabin may have an influence how about so we talked about the gender the age of siblings and spouses any idea what if so if you have more people with you from your family are you less likely to survive or more likely to survive yeah so for most decent people I think you are less likely to survive right I think and it's it's not a really not judging someone but I think it's it's usually the common sense common sense is that most people they but there are cases right so then maybe a family goes so right let me think probably that can be a predictor right how about children is it going to be impact their ability to survive possibility a chances to survive sure there's a possibility right and you won't be able to do all of them I will just show some visualization there I would take it number okay let me go back actually how come the passenger ID so one question is goes on is the passenger ID does it really matter mm-hmm okay so in this case I think passenger IDE it's just a number right it's just there but really I mean think about this let's say if this passenger ID is correlated to the cabin so people in in some scenario the people who came first they landed in the lower deck versus the upper deck right so there can be so many things but in general I think the identifier is the goods and IDs they usually are student IDs these are not useful predictors but you have to be watchful and careful about this you have to be really cognizant of the fact that you can make a mistake and then just how about the name automatic ticket good name the could may be a protector and actually I want a show of hands I think I think of just just to maybe see more energy there right so how many of you think name could be going to have an impact on survival very few people and is it fair to assume that people who are not raising their hand think that okay yeah you have a question yes sure Sony knew you can actually depends so let's say you didn't have a gender column there first for whatever reasons right if you didn't have a gender column you would actually derive the general from name there is a possibility right so you will see by the generator boss and what if the name has sir or general or something right so you you never know right so they could so neem actually can have some clues into whether whether it's right or not I'm not saying it's the case here but just to give you a general you have to really think like a detective here right so just keep thinking that what is it that you can use okay coming to ticket number based on the discussion any thoughts I'm sorry yeah what was the fishing so I thing that we talked about her to attend to quit have an impact right so definitely change it will have an impact and even now I think it should be the I'm so somebody actually in my last cut somebody said that I think it's not the case anymore but as it turns out so now if any incidents like this happen it's not in the manual but if I survive most of them which is which is sorry and unfortunate but this is how it is right but but in this case Titanic is known to be in okay so number ticket ticket may have some indication whether it may represent I mean it's a first-class ticket or some certain cabin or some some privileged person right so sure I mean it can it can really indicate that apology I mean how things were distributed in the cabin so ticket to me having some clues but really you will have to know the features maybe a has some meaning a size 5 we see or maybe only some subset of the tickets yet in here most likely I mean if you but if you have to make sure that you so and then okay and similarly so actually so it's only cabin which is where it is obvious but there is some other data I think some ages are missing some genders are missing and then me so there are some missing values so this is a it's a it's a little easy dataset but it's it's really good enough to actually get started it's not like iris dataset which is very well behaved and then you just plot whatever you wanted one so but that's a good point right so you have also have to you also have to look at the missing value for the cabin yes you have to you have I I won't be able to do it I mean so it's so whatever I'm if I get into that direction it will take a lot of a lot more time but I think you are going to be by the end of this presentation you are going to be armed with all the things that are needed there right so you can go home and then try I mean not all the code is going to be there but you can really when you go home you will be able to reproduce some of the things and actually add and experiment a few other things yeah so one is a very strong correlation like it's like the same right but you wouldn't see that strong correlation but if it's if it's strong enough then you would make you worry about that okay so you put cabin I think perhaps CNA 5mc and 123 or possibly P and B and other things so they they may not make any sense but what if you pocket eyes them into a B and C and D then you can extract features out of that and that that's a good practice I mean you should always be looking at always be looking at how to expect more features for instance age so you may not see age to be a good predictor but what if you threshold that and whatever the legal definition of the child goes at that line and then you create another feature you say anybody who is less than this I will mark them I will add one more column to that row and then I will mark that person as a child otherwise it's an adult right and then suddenly you have another feature which may have been lost in the age itself right so if you have all the edges it may not be obvious completely but as soon as you split that into adult and child it may become yes so you sometimes that you have the data that you need but it's not in the form that you should have it so you you really need to actually you need to modify it you want to extract each other with them okay and it comes from really it comes from intuition when you start solving problems again in the gap then you realize there are different techniques right so you can if it's a continuous value you can replace it with median mean you can whatever if it's a classification problem the average of that class you can completely get rid of the rolls it totally depends on your situation if you have if you start moving the rows with missing values and you end up with 10% or 5% data you wouldn't want to do that right so you would it it is really how you what is your situation and whatever what is your unique situation there's no single answer to that okay so the question is if you have missing values what would you do do you remove the rules or you do something else so the answer is it it depends if in many cases you may be able to replace the missing values with the median value or the mean value or the mean value within the that particular class or you may want to remove the row right so there can be many different techniques you wouldn't want to remove the rows if there are a lot of missing values here and there right so you don't want to end up with no data at all right so because but whatever is the column is a good predictor right so there are values that are missing but with whatever values there are it's a useful column sure and certainly I mean there are many things that can be that can be done there and so missing handling missing values would be actually a detailed discussion goofy yeah that's that's a good question right so and this is usually the case right so think about Amazon when they are suggesting new products or recommending your friends or doing any recommending connections for you it started 10 12 3 key 40 features problem the they go to the finest level of granularity that they can get the data on right so did you come back in last six hours or last 10 hours or last 24 hours last week did you are you a holiday shopper or not what's your zip code what's your address which city are you from sorry can you imagine I mean so those companies just take an example they literally would have thousands and thousands of features from where they will have to extract the most useful features to build their model who should they send out prime offer who should they send out a kindle all right so who should read this book and not that book so all of this actually requires a lot of research and and that's why I really I mean doing particular modeling is not it's no big deal but doing it on scale that is where things become more interesting with more challenging and that's these are actually that is the problem precisely that these companies have solved and then they are okay and the last one is the port of embarkation so I think they were there are values there okay so what is the data type of each column so I am showing this because I think this can be a big pain point when you go back home and try to do it because it turns out so L apply a Titanic classic th guess gives me the class of each column so passenger ID is an integer and fine with it but survive is also an integer to be really think it's an integer it was 0 and 1 it's really a label I mean a we shouldn't call that an integer but our tries to infer based on whatever data was given and it thinks it's an integer but when we are doing a classification problem we have to explicitly cast the target variable into a if it's a numeric variable or integer variable they have to convert it to a categorical variable just to make sure that so it turns out and then there are others as that so when you get this I do this you will see all of these labels here and simply you can simply cast each of the this particular target variable just by saying as dot factor and then it just converts that to do yeah okay any thoughts you did are do we think our hypothesis was correct yeah no we didn't get the right so right now it shows what was the proportion of people who died versus survived so more than 50% people died but we it just shows the proportion right so how about we add some more and more you know some numbers and positives to it right so this is not the code that you would use but this gives you an overview ready what's the proportion of people who survive and versus the people who died okay so this is just to get you get accustomed to the dataset okay and this is just another fancy so this is a package but my Fuji so if you are presenting your data your findings it's always a good idea to visualize your data nicely okay so let's come to this we talked about this we think the gender that gender is a good predictor and this is all the this is some code that will create a pie chart of gender and whether they survive and again you can replicate this using the convention okay what our hypothesis today so all about more than more than 3/4 of men died and more than around 3/4 of the women this one right and this includes children but I have female in here let's call it that okay so now if I had only one feature here if I build a model that that just says if it's a male I mean I know for sure it died right so it can be a very simple classifier yeah generally puts in here plus equals died else supplied right so this is my machine learning algorithm there right just based on this inside and it's going to be fairly accurate and how many of you have ever played the kragle at all so I didn't realize that I would have otherwise I would have shown you I mean it's very easy to create an account and then you can just create a model based on gender and then upload your prediction then you can see that it's it's fairly accurate just based on gender yeah so it turns out that there was the case right so there because there are some missing values so we don't know why those values are missing right no so this is all explicitly only for the agenda that was identified I didn't actually look into that so that's a good number for you then it's just I mean what you will do is you will just replace with this with appropriate filter select gender is not equal to male and not equal to female so you're gonna you will have or I think it's you can simply look for a missing value and that should also give them all because there are some missing values there okay yeah but that's a very good point I mean we should actually look at ICAST though so so this was a good enough particular there how about age we agreed that it would be a protector right so so I'm using this summary of age summaries of its function in R which gives you a I think this is seven point somebody here so it gives you the minimum age and so they have ages in this I think they're somehow normalize the age in this fashion and you have the first part is so first quartile is about 25 percent of your data so 25 percent of the people were under the ages of 20 the median age like right in the middle was 28 the average age the mean age was 29 and then third quartile third quartile is 75% so 75% of these people were below 38 and there was someone who's eight years old is that right and then there is the missing values I'm in 77 okay and we go to okay so what about what if he but what if you segment the age by survival right so we want to look at the summary of the ages only for the people who survived and only the for the people who didn't survive right so I'm trying to segment this in see is there any difference in age in the distribution when it comes to whether the survivor time right and I'm going to be using box plots that you have seen right so on the left so we have so this is actually the pH distribution I think I should have let me actually show you I think I again didn't copy and paste this okay so this is the people who did not survive and then if I just to do changes to one any insights or I think and understanding that there's no insight here is an insight itself right okay that's been the youngest person actually survived yeah that's that's a good insight right so can everybody follow this right so a bun inside that I didn't notice that right so one observation here is the people who were the person who was the youngest on the ship and the person who was the oldest on the ship both of them survived because the min and Max are there and the min and Max for the people who died any other no correlation okay so because the median age in both cases is about 28 the mean ages they are close I mean yeah so the people who are slightly older they died but but it's the median of that particular crowd right it may or may not change right so I agree right but it's a median of that particular processor it may change and we have the third quartile in the idea having slightly older people there but really I mean there's not much here it doesn't tell us anything but we we had this hypothesis we said that a chasm has an impact on survival right so what's going on here okay okay so any any more specific yeah yeah that's entirely possible but we just we discussed this our hypothesis was that age will have an impact right but over here we see that there's not a very strong impact also that's that's that's a valid point there is a lot of missing information okay what else yeah exactly so what's what should we do here okay so do you do you think if we have if we because when we say age has a age will have an impact we're not talking about any age we're talking about children Yeah right so gender was there but when we talk about age it's not just any age but when when you're looking at this it's just it's a aggregates right so we have to find a way to actually separate children and adults right and then we may want to create another column or another category that actually sets this threshold after age is less than this then next guy versus okay okay so we looked at that and these are again some box plots and the left and then left one leaving that for you to just just think about this app for your mobile but this is the age distribution by gender this is the age distribution by gender right so on average men were the median age for men was slightly higher than women and the circles on the top they are outliers so there were some very old men as opposed to women so women were relatively younger pac-man there were some very old man and this is those bubbles actually signify outliers there and then we have a distribution by survival and it looks like the median age for surviving people who was was much lower but as a homework because we still have a bunch of things again trying to build a histogram try to look at the density there and so this is all that you can do at home there's some some interesting insight here okay so we have this done color density plot so this is the distribution of Ages by survival so the red one is the distribution of people who died versus the green one the ages to be some people who survived any insights here does do the to look different okay yeah so this is a PDF right so it's it's a smoothing function that would take you to the to the less than zero right so that's a very good point and good observation I'm happy that you were paying attention but really what happens is that when you are fitting a density function you can have a lower value it just signifies that okay so yeah it is a density inch can't be less than zero up so they can be values that are less why would you want to do that because I mean it's so you're here so you are actually impacting the way the smoothing is going to happen right so you are going to ship this this key piece of information to the right so this is somewhere over here right does it tell you anything there is a fist bump there what the soul shows is that this ages this is high probability agents only there's a blip and then it goes these are kids most likely very little kids right I don't know what age matter it is right here and again we can always lice and I see now it can be an artifact but that bump is there right so bum bum clearly shows that there was yeah so that that is an artifact right so and we agree so that the negative value is artifact but that bump is here speech from data and we don't want to mess with the spoon in function because you will ship this okay so this actually is some that eh had something to do with wet survival okay does it make sense you have to keep going so this is code for data splitting and I'm not going to go through this again but whenever you are a novice mistake when you're doing predictive modeling is when you you clean the data on the data set and you test on the same data set so it is going to work right but it's a good practice to be randomly partition your data into different partitions and you test on Ukraine on one partition and test on other partition otherwise if you are it's like you're making your algorithm memorize the until this is just memorizing your data completely you are just going to keep on training right so so this is the code for partitioning so I'm going to just try to just be very quick we have so now so this is ggplot2 this is a very nice graphical package and the dataset that I'm going to use here first is due to interview service the basic things and then I will come back to it again so this is diverse data set and so this is the carrots and the cut and cut has I think five values the color the clarity get cable icon or cable there's no these are some numbers in the director so what would have an impact on on the on the price for instance right if you are trying to predict the price okay so that's it this is a histogram look any different in terms of like to be that histogram table so this time I used instead of the base extra grant function and using the GG plot and the be GG plot is is so this is actually so ggplot actually renders things in layers so so this is the first layer what I'm saying what I'm telling where my function is that I'm going to plot this data set and the next layer is saying that I want to build a histogram and it's visibly better than what what we do right from an artistic landscape it will look that nice similarly if I go to the density density function so this is the first layer here and again say in debt this is this is the plot for this data set and then this is the second layer which is telling it that okay on x-axis block to the kerrick's the values are going from 0 to 5 and then fail at the great other right then it's once you start giving you that if it's two people and this is a this is a scatter plot for cats and if I look at this so again this is the first layer here and first they are the same thing but the GG plot forgiveness of diamonds and on x-axis plot carats and on by axis the price any correlation between character and price Thanks so it arbitrator the freezing can its think but so the price is roughly on the higher side but not just wrong but what if I want to know what if I want to know the correlation by color you remember there was a color term of column India right and so this can be misleading here because of this color records to the column and this is a parameter so there were different colors we don't know what colors goes well but these are values right so now we have been able to work them separately all of the whatever colors those were right so the color d e f g h i j the people even do segment it even nicer right so what this is what we are doing we are trying to do is that we are plotting all the segments separately just just slicing and dicing the data until we understand until we get something out of it and it's just literally just two or three commands that i'm using and just installing a package and then you we are done okay can they be more yes even more so look at this it's beautiful right I mean so you have to get it and your price you have been in each category you have I think feared good very good and p.m. an idea was the quality of cut right so and then there was another variable I think there was some there was some in protecting matrix symmetric there so now I mean you can if you look at this if I look look at the first row it's basically a four if you look at it it's support initial block right I mean you are adding more and more dimensions here and you're slicing it actually beta to the greatest detail that you can write okay and turning it again okay so what do we have here we have if we look at this here so what we are doing here is we have Y a sphere right so this is here and X is the class so this is the class and this is the classroom and then we have we are doing a box plot which is for different ports of embarkation is it different and the first one is actually missing value if you look at the first one there is no value here and this one is with curved edges and this is something Southampton do the look different visibly different right so we can see that the people who bought it from so there were few people from Queens birth and then definitely the people who were from South Bank and I think it looks like there was a lot of outliers here and I don't want to get go into the details I think we are short on time but it just gives you that idea that how how mean you can slice and dice data this is it so now it is fair and survived right and I actually created a column adult and child and then male and female look at this I mean how how many so this if you look at this plot this is basically whether the person died or survived only for here and only for children right so this comes it's made child survived died or survived and then again that is for male dog died or survive and then you can really slice and dice data there okay and there is just try some variation I mean of you would be surprised how I mean once you start doing this different variations you will see how how insightful and how valuable that is so yeah so basically all you have to do is that with the blue here so replace any numeric value here because we are doing the Box block and type any categorical variable and then any can about okay categorical variables here and try to play with this and then you will see how package will start emerging there okay let me actually directly I wish I had more time this is so so I'm using the package called random forest so it says it doesn't could not find a package for RS okay so follow my desktop I guess so so really let me I think it because okay okay okay so I'm going to make a point here because we had some observation observation here so I want to actually make a point before we move on because this so I can it be okay so quick point to note here is you guys remember that so what does show this how many are there sit over classifier that's two doses so exactly how many producers were classified and virginica particular zero and go Jenica but if you look at how many per sequel never classified as for C colors it's 47 and there were some of course equals classified as virginica and at the same time for of the materials were classified as multi color and then they said some of them were classified as wrong class does it ring any bell an int it did we did we notice something that we were analyzing features in fact we saw some overlap there in the distributions so definitely these two classes are likely to be confused with each other okay what I would do is I will send all of you the slide deck I really quickly have to show you once you ramp up on this you can actually try to do it so you can use random forests for regression or for clustering for getting the importance of the variables you can do that so all of the data is there and then this one is a neat one what this shows is that you can actually also get in variable importance how important getting the classification and what this shows you is that so these are two different criteria for getting the variable importance and so this shows the first petechiae says the weight is most important in second one says that the displacement is most important so it actually turns out that this intuition was correct and then there is some other data that we have and some resources okay so I will quickly jump to so this is a new tool that Microsoft released recently in its it is me ready so what you can do is this tool actually lets you upload your data set it's a machine learning of the service so you you want to do some do some classification or regression or any other machine learning tasks so it has a bunch of different algorithms which are already there right so you have so this is some of the standard data sets that you already have you can play with these data set you have different for data format conversions you want to convert data to different ages format you can do that data input output you can create and you can really read data from any web source you specify the URL and you have the data it's loaded then you can do different transformations of data different kinds of filters and I love this thing because it's you I I did the splitting and partitioning code I left it here and the ugly vote that they are there just to just to convince you how to reduce for best thing yes because this thing just need me I drag and drop a partition sample here I will connect my dataset to it and I choose what kind of sampling I bought and what kind of what rate of sampling and then if I let's say if I choose 0.5 and then I run here and it is I don't know where the machinist it's a high compute instance and it's running on the cloud somewhere and it will give me all the politics really that I need absolutely absolutely and I will get to that this is and this is this is my dataset after partitioning so I have 146 rows into our balance and before but next thing I had 891 process 50% and right I'm done and I'm using a decision to cluster this random forest but there's a bunch of other algorithms so you have been chopped she running and there's a long run and we don't have to do anything you just define parameters and yeah worst thing ever you have regression algorithm I mean it's really a lot of things and then even if you have a big data set yeah what is that yes yeah so there is something called the our module so really what you can do is you can create your own our module and you can execute our step aspect and you can create the module and then you can nicely define the input and outputs and then you can create your own module ready now and I heard the news that the Python is coming yeah yes so you pay by use I don't know but I'm using I'm using the 44 and I don't use it a lot in the sense that I the the free tier is enough for me so I wish it was you can download you can dump the data said you can say with the other data set you can save the output and if the neat thing is you can publish this as a web service within 2 flex right you built a model and now we can let anybody else and I think I have already scheduled that talk and only I will be doing this this is neat I mean it's you can think about this as Apple the way they disrupted mobile apps now this is something that anybody can build a machine learning model and then publish it and then really make money I mean if they are good enough they can just really publisher anyway so documentation is catching up so I think that is one thing that needs an idea but eventually it will catch up and if you are if you know if you have reasonable background in this you know name it is not actually I think you will be able to so for instance this is my Titanic data set right so I just created uploaded that he does it and save that created a uploaded the CSV file and community Edison and and look at this so previously what we were doing was we would run in all of those commands to get there get all the box plots and histograms right so right now there is only two options here so I can get so histogram for survive doesn't make any sense but let me some other page and so let's get so this is the distribution of age and I can also get a histogram of age here so we get a histogram here and then you guys can really explore that another neat thing about this is I copied this they stick here okay so this is I'm creating another experiment here and when I created this experiment I want it to be a bit different I don't want for decision don't want for decision trees I want six decision trees I just changed this and so quickly I was able to replicate the experiment with and then what I will do here is I'm going to so this is what is shown me so it's cute right now it's so it will check out the machine somewhere in the cloud for me it's I don't know what's underlying logic I mean how do they decide if I need a really neat high compute instance or not but so right now now the model is training and did anybody notice in the new run there left side is on the jet check check right if I have to run my code again it could run again no matter what right so right now these decisions are saved the left what if I'm already saved in the cloud and it didn't need to run again and now I have this here and this is another one I just freed another I created and evaluated into the model you have to specify what you've got McCallum and how we get to that that's a good observation bracket so this shows me my rocker right and usually I will be calling another function coordinator all right so this is keeping your rocker so this is random guest and this is the school dataset and this is precision recall and this is left so there is all the metrics so true positives false negatives false positive positive so this is the confusion matrix the accuracy the precision the F score the recall I mean it's just happening within the plane I can also compare this and this will actually compare which one is better I can do that and if we plot the Stupak curves on the scene on the same scale and then put show me because I can immediately care which rocker is better because I can Nixon no so for me I think that if I recall correctly because I'm in a program called business fraud now but if I remember even before that this is do you know what's surprising for this I think hundred and twenty five to two hundred and twenty five dollars you can get a credit but I think my leverage is higher because I'm my limit might be 200 because of being in the business so basically if you are a start-up okay so any static would get that but for individuals I think it is $135 because I'm I don't work for Microsoft so I don't know so you may have if you are your school may have a partnership with them so you can check your school of course so I can be having my work space and then what I can do is once I'm done I can go ahead and save that work space and as my production work space and then expose that as as a web service to production you mentioned that you got some slides house half a school I don't post it on the Meetup so they say yeah so I think this is not going to be posted on slides this is something I can maybe send the URL go sign up for a free account if you don't use it you don't pay me right so just go and sign up Oh unless you get off a job in this I will the PowerPoint shows us now or you sell your email so slides I do it - we give you my my business call something a war no I'm not going email it's going to be uploaded on the meetup page oh don't be the yeah and then I also send out an email to meter reader page that's also that's what good yeah because I mean I don't want to spam everybody right tell me if you need it you can download the slides otherwise so coming back there so your production right so so you can deploy that as needed so it's both for your prototyping and so let me know if somebody is waiting outside because this is this is actually some look at the samples right so you don't know right so you don't need documentation really here if you have these samples I wanted to show this thing I this time I'm performing the scene right the rosters are overlapping here and then we have the precision and recall for both of them here the accuracy and then a bunch of other magnets that are there and I was actually going to the samples so they have a bunch of samples here so some regressions and examples some binary classification example let's so so this is a very well known Peter said it's but you see again from UCI depository so they have missing when you scrub alright so and you can you can choose a custom substitution value then right so you can like yeah oh okay I see okay okay yeah so I saved this and now if you look at this now I can do this with mean I can replace the PE and I can click mode the movement diagram so to get me right I can do this I can do a whole bunch of other things here right then I have this basic problems I have I can split the data I can initialize the model and train the model let's go to the model there is a there is some there is a recommendation here some recommendation engine here example yeah we'll be recommended so that's there and you can see that again I have some movie dating periods and then again split drink recommender score scores recommend dinner and then in three minutes once it goes out once so really I mean if you want to get started and you are do some machine learning okay so yes sorry I mean the last time I take what happened was items I try to go slower in the beginning and then if you are really interested in this slide I think I did a food in Santa Monica I said how can you go if there is any messing let me know and then this this is going to be a there is going to be a bit of cleanup any other questions about