Introduction to Azure ML and Cloud Computing

Key Takeaways

yes thank you for reminding me oh that's a lifesaver okay so the idea is uh I don't care how good your algorithm is right the the idea is the more data you throw at any algorithm the data the better the predictive model will become so the the idea is you need lots and lots of data okay the next thing need that needed to happen was computing power so if you have to recursively C calculate a million ukan distance a second it requires a lot of computing power right it would probably melt on your own computer or your own laptop so the idea is we've move we've moved basically to the cloud to utilize this extra power because what does the cloud have the cloud has a gigantic resource of computers lots of computing power for us so it can handle this surplus of data okay so let's talk about the different machine learning tools out there all right so let's talk so most of you guys have probably been in the realm of uh Excel so let's type so Excel so you can actually data mine in Excel you can actually build things like neural networks in Excel but the problem with Excel is that it probably it starts to melt your computer and your laptop around 44,000 cells and that is because Excel has a lot of overhead right each cell in Excel it's carrying other things with it like what is the background color is this underlined is this highlighted what kind of format is it it's carrying lots of different things with it and that makes it very very bad for machine learning okay um so next week we'll teach you uh programming languages that that tackled machine learning so we'll we'll teach you R but you can also do uh Data Mining and machine learning in Python so that's another popular language so I'm more of a python developer than an R developer myself um but just so you know the the rising popularity right now for data mining and machine learning is currently in r okay you can also do things like Java and things like that but it's not as widely uh used okay uh there's ALS if you're also from a university standpoint uh you might have used a a statistical language called matlb okay um but there's lots of what's called uh cloudbased machine learning now right so you have players like big ml uh Azure ml okay AWS ml right they all have ml in their name okay so so lots and lots of tools out there and and a lot of people ask me which one should I use well the long story short is you want to use whatever your company is using right so if your company's on AWS don't go out of your way and try to pitch Azure ml right unless you have to so I would just learn whatever it is that uh suits you so for the purpose of this course we are going to choose Azure ml okay so Azure ml is because well it's if you choose one Cloud stack you you have to go all into the cloud stack because the cloud Stacks do not talk to each other very well they integrate very well vertically but not horizontally if that makes sense right so let's take a look at one of the the the players uh big ml real quick before we get into Azure ml all right so big ml okay and actually just look at the Titanic uh so this is a there is a a public data set out there of the Titanic passengers and what that is is uh it it's a record of whether or not they lived or died how much they paid when they got on the boat what was their agender and basically there the socioeconomic status right and this is a a model that came out of a program called Big ml it's a cloud-based machine learning platform and this is one of the trees that is spat out so it's a decision decision tree right so if if if you're if you're in first class and all that good stuff and you go all the way down and if Factor filters out your age then you for example live equals true okay so this is an example of a predictive model and you notice that it's very pretty it's it's a very drag and drop uh kind of uh interface now the reason why we don't use this one in particular is because well the thing about data mining is that you need you need the whole picture you can't just have machine learning machine learning is a very small sliver of that entire pipeline right because your data is piping through you also have a data warehouse you have a data storage you have data bases right you need your machine learning platform to talk to all of that and big ml as it is it's basically just a standalone platform right so the idea is at the end of the day you need the entire platform to talk to machine learning so that's why AWS machine learning and Azure machine learning are preferable because you can hook up those things to things your data process engines like Hadoop you can hook it up to your data in gestur you can hook it up to your data warehouse storage and databases okay so that's why it's important to have the entire picture uh together and what I mean by the entire picture is the entire stack uh all working in unison okay all right so uh today we're talking about Azure ML and next week uh Brian Johnson will talk to you guys specifically about our programming so this is going to be a hybrid class uh when we use machine learning we're going to use uh R and we're also going to use Azure machine Learning Studio uh to get it done now we're going to teach it from a perspective of uh Theory based so the idea is you can take the theories and the steps that we would apply in our or asml and take it to any other stack such as python mat lab a USS machine learning and you'll be fine okay so let's actually get into Azure ml now so I know not all of you have an account and that's fine uh this is more uh supposed to be an educational uh webinar and I'll give you homework uh after the fact so you guys can actually start practicing uh Azure ml so uh I will just simply go to Google okay and just type in Azure so Azure and Azure ml are tied together so what it means is you need to have an Azure account uh so I'm just going to type in Azure free trial and to do that you would just click here uh to get into an Azure free trial page so click try now so when you click try now it's going to ask you to log in right so once you log in it's going to ask you for a bunch of things like your your your your information and your credit card number and things like that and don't worry it's not going to charge your credit card number unless you're uh from I think China and India it will charge you a dollar uh for the the uh I think it's the foreign look up fees or something like that all right but other than that you'll be fine so you once you have a free trial account uh you will simply go to uh you'll go to manage. Windows azure.com okay so manage. Windows azure.com takes you to this page so this page is the portal so the portal has all the cloudbased apps that you can access so this is a platform so Azure is a cloud platform so you can build in an entire for example Twitter or Amazon or anything you want basically in this entire stack all right see this is true of AWS like all of Netflix is built on AWS okay and this is true of azure as well so you can whatever it is you want to do uh business-wise with it and infrastructure uh Azure or AWS can be that infrastructure okay so you notice there's a bunch of apps and specifically one of these apps is machine learning okay so uh there's way more tools databased uh data typee tools uh in the cloud than we can cover but we will cover basically the basics uh at the at the boot camp so we'll teach you things like H doop uh we'll teach you Azure ml we'll teach you storage and things like that so uh to get started we need an Azure ml workspace and if you click on if I click on machine learning you notice that I have three uh workspaces here okay now uh each workspace will need its own storage account um but if I want to go to create a new workspace I will go to new okay and I will go to uh uh data services machine learning and then specifically I'll go to Quick create right so quick create will let me also go uh and select and name my workspace so I can name it uh uh fuk workspace okay and then I will set myself as the owner of the worksspace and then I will give it a storage account so what does the storage account do well this is where you're going to store all your data your models basically everything okay so you also want this to probably be in its own dedicated storage account so in the future you don't accidentally delete your storage account asso of your Azure ml workspace or else everything will be destroyed okay so basically I'm going to create my new storage account and I will call this fuk workspace as well so that way I know that this workspace is tied uh or this storage is tied to this workspace okay so I'm going to go ahead and create uh this workspace okay and let me know if I'm going too fast okay so this is how you make a works space so once you make a workspace it's going to create you an Azure ml uh workspace in Azure okay and this will take about 2 minutes to set up okay and let's continue on with our machine learning tools so a lot of you will probably also come from a SAS background SAS okay so SAS is probably one of the most powerful uh data mining platforms out there today and it's probably only the oldest okay it's really huge in the medical field and things like that um but the problem is it costs about a million dollar okay uh a license so only basically companies can afford uh this level but we're but r at this point R and python at this point are coming into fruition it's basically it's at the point where it's had so much open source contributions to it to the point of where it can take on SAS just fine and the cool thing about open source it's free right so uh so now there is a gigantic decline fine in SAS and and a huge uh increase in the popularity of R in Python okay okay so this workspace is now created and let me just delete this one real quick okay so I have a fuk workspace here now and if I click on fuk workspace it will take me to a dashboard and there is a signin button right here so if I click sign in it will let lead me to a place where I can log into my workspace and you will notice that some of you uh if you're from Microsoft you might notice that this is the old portal uh there is also another portal to access Azure so if actually if you just Google Azure uh you just type in Azure microsoft.com and click on Portal it will take you to something that looks like this this is called the new portal you currently cannot create a machine learning workspace on the new portal okay so make sure you go to the old portal and to get to the old portal you have to type in manage. Windows azure.com okay so that's how you get to the old portal so we will won't be using this new portal at all for the boot camp okay it's still I don't think it's fully ready yet even though it's fully released and out of beta I don't think it's ready uh uh stability wise yet okay so let me so if you it it should automatically sign you in to this workspace and you'll notice that I'm in this workspace now so there's a few things you should look at for okay so there's home okay at the very top there's home there so these are just navigation bars right there studio so the studio is what we care about right this is the canvas from which we will build our experiments from which we will build our models where we will munge our data right um there's also a gallery right so uh this is their basically their their if you if you built a model that you really like you can submit it to public Gallery so this is kind of like the GitHub uh of of uh of asml so if you like something you can submit it and then teach other people how you did it and then they can clone your experiment as well right so I'm going to go back so let's look at a few things so the the studio is always what you care about you always want to be in the studio um a new button is going to pop up when you first log into Azure just close it it's it's fine uh so these are experiments so uh it a save file inside of azure ml is called an experiment so just like how when you save a Word document right it's it's it's a word it's a word file or if you save an Excel file it's also called a spreadsheet right so uh this an experiment is just a name for the saved file of your model okay then there's the idea of web services so if you've hooked up a web uh uh if you've built a model and you've hooked up a web server and now this is where your services lie where people can call your service uh on the cloud and let me show you an example of that so if you go to uh demos. datascience dojo.com so I'm just going to paste this really really big okay so demos. datascience dojo.com you will uh go to a list of our uh demos okay so if you if on the very bottom there's a Titanic survival predictor okay so this is an example of a web service right we've built a predictive model that predicts whether you will live or die based upon your circumstances right so whether you're first class female and what port you embarked from and what this is going to do is it's going to call on the back end it's going to call a web service and the web service is going to basically just score your model is or uh make a prediction right so based upon my circumstances I would have a 90% chance of survival right so by the end of the boot camp actually by probably by Wednesday of the boot camp you will learn how to build uh the back end of this uh app uh we won't teach you to front end right because that's just HTML CSS JavaScript but we will teach you how to build a model that that basically Powers this this guy right here so whether or not you'll live or die uh based upon your circumstances based upon historical data can we predict the future okay using machine learning okay so this is where your web services will lie okay so it looks like uh by just changing my port I decrease my survival rate by 1% okay all right so let's go back to this so uh there's also notebooks so notebooks are the idea the idea of notebooks are actually uh enclosed environments right so uh if you if you know python uh this is what the notebooks are for and if you're more familiar with AWS machine learning basically all of AWS machine learning is notebook right so if you're more used to just pure code you would build build yourself a notebook and then there's the idea of data sets and your train models so train models are basically your machine learning models that you've you've trained on your data right and if you go into settings uh there's also the idea of uh authorizing other users into your workspace so azl is the closest thing data scientists have right now to a Google doc so if you want to work on a team on a machine learning project or a data science project or an ETL project or a batch processing project you would actually just make yourself a workspace and invite all your co-workers into the workspace and then you can work on it together I think that is phenomenal okay all right so let's Jump Right In so uh we will now create an experiment so you click on new uh and then new will let you create a few things so you can import data from a local file right here right so you can import from a local file and choose whether it's a CSV tsv and whether or not it has headers or whether or not you want to overwrite a existing data set okay um you can also uh create a new experiment so I think we will do that now so you notice that we can create a blank experiment or they have a bunch of other uh example files you can choose so if you want to do if you want if you don't know how to do things like image recognition uh then you can just simply search in uh image right or actually I think it's called digit recognition but yeah I but you got the idea all right so the IDE is we we'll just make a blank experiment so a blank experiment has basically nothing in it right now and and uh so this is actually the workspace that you'll be working with okay so if you look on the left these are all our available modules so modules are basically tools or functions that we can use and on the bottom you will see a bunch of controls right all right so if you click on save data sets it already comes with a a bunch of sample data okay so if you just drag in a a data set you notice that it's a drag and drop kind of uh usability right so if you you can drag it around so the entirety of the data set is represented by this node thingy this module okay so these are called modules so you notice that we don't care how big or how wide or how many gigabytes this data is it's just it's just encapsulated so if you ever worked with things like data flow diagrams this translates very very well to data flow diagrams so all the outputs are going to be on this node right here so this module right here uh if you want to click if you want to see what it can do what what this node is outputting right so this node is outputting some kind of data right so if you Mouse over it it says data set okay so that's the data type it's a CSV so if I Mouse over it and I right click on it I can visualize so you notice I can visualize the data so you notice that this is not too big of a data set it's it's a 205 row data set and it has 26 columns okay so this is uh automobile pricing data okay so you'll notice that yeah the entirety of the data is Right Here We can load it into a notebook too so if you're more used to python or python 2 or Python 3 you can do that as well so if you're more used to AWS machine learning this is exactly how you would do it um you can also download the data set so if I click download it will give me a file where I can download the file so that may or may not be preferable because remember remember why we're in the cloud in the first place we probably have too much data and it will probably will not fit in our laptop TOS are our computers so the idea is if you're probably using this you've probably outscaled your computer already okay so the idea of working with big data okay so now that we have data what do we want to do with it well we can do lots of things with it um we can uh there's lots and lots of other functions we can do here so let's look at format conversions okay so if we want to save this data set we can just drag in a convert to CSV module right or if we're not happy with CSV we can save it to as a tsv right so if I connect it right so now I have two workflows right so you see one workflow going into here and one workflow going here so on one side I'll get a CSV and on one side I'll get a tsv so if I run this right now right so uh you notice I have to run the the experiment to execute the functions because right now these are just blank instructions so once they have a check mark it means that it's there's something in it there's something cached in this module right so if I right click on it I can now save it as a CSV or I can save it as a a tsv so this is going to become very important on on Thursday when you guys will create your own models and output a batch processing file of its uh predictions okay now you can also do other things too you can also do Ingress and egress uh which is a really fancy just a really really fancy term for reading in data and writing out data okay so let's let's see what this skin can do okay so I can read it from a blob store so a blob storage that's just really fancy name for just cloud storage so if you've ever used Google drive or one drive or Dropbox this is that but on steroid so Azure blob I think by default comes with 200 terabytes um uh and it's I think it's a one tenth of a penny for a gig for a month so you can't really beat those rates all right so AWS machine or in AWS this is uh it's called S3 so if you're more used to S3 uh this is the same thing okay so you can also connect to a SQL Server um so if you're more used to SQL right and it's in SQL you can do that as well uh you can also read in from a hive query so Hive is uh Hadoop right so if you have a Hadoop cluster running well and you have a hive table well now you can quate the hive table directly without even downloading a file and re-uploading the file because that takes some time as well or you can read just straight up from a web from an https if you just do ww. I think CNN here it will just crawl and scrape CNN.com okay but if you if you want to just read in data so I'm just going to go to Google and Google UCI data set Iris okay so this is a Hello World data set okay so all I do is Google UCI data set Iris okay and this takes me to the iris data set so this is the hello world data set of machine learning and if I actually want to look at the raw data so I'll go to data folders and then iris. dat so you see that this is the this this is the raw data just posted on a web link right so I can just copy this web link L right and paste it in to this HTTP crawler and then it's a CSV and does it have headers no it does not have headers so I'm not going to check this box okay so now I can just run this model and it's going to go and it's going to crawl this web page right here and it's hopefully going to parse this data for us okay all right so let's look at that so this is going to read it in uh it's going to create us a schema all right so uh whether this column is numeric and things like that all right so it looks like it readed in so if I right click on the output I can now visualize okay so you notice it didn't know what headers it are so it's just made it column 1 2 3 4 5 right so and look our data is now loaded in to Azure right so and we can now visualize our data too so we can see that there's there's uh equal numbers of Iris vers color Rosa and virica okay so there's three unique values and there's one missing value right and I think the one missing value is is an extra row that got pulled in and it had nothing in it okay so now that it's in here right I can save it as a data set right so I'll say this as Iris Ingress or something like that right and if I save it it's going to save it to my workspace so you notice that if it's a reader module it's it can't cach it so so if I run this again it's going to go ahead and go to this website and parse this all over again okay now that's going to be really useful for a web scraping perspective because web pages change often but this is a data set right this is a static an internal data set it's not going to change so it's going to be a waste if it's going to parse it in every time so that's why we're saving it as a data set right so if I look under my save data sets now I will have an iris Ingress that I just pulled in you guys see that now right so again I can also just convert it to a tsv from a CSV and just run this again right so this is going to be all right so this is uh so you guys notice that it's a drag and drop uh version of machine learning okay so now I can also write data right so I even though I just write in data I can write data out to other things as well right so for example if I want to write it out to a blob Star right for long-term storage if I want to write it back into Hadoop or into Hadoop I can do that as well so you notice here I can do some crazy things uh from an ETL perspective right I can write in from one Hadoop cluster and write out to another Hadoop cluster with Transformations and maybe predictive modeling in between okay same with SQL Server okay so all right so let's what else can we do okay so let's look at this uh so let's look at some things that we might want to do so if I drag in this automobile pricing data okay and I visualize it so there's lots of things wrong with this data set there's a lot of things that probably need to be cleaned right and that's just called pre-processing right because machine learning uh models ex only accepts clean data so every machine learning model out there accepts clean data and how you clean that data will determine the success or fail of your model okay so let's take a look at some things right so let's look at symboling okay so in symboling yes um okay sorry to interrupt um I'm not sure if it's just me but I'm having a hard time following um what we're trying to do are you trying to give us an introduction to all the features of the Azure ml studio and um as a pre- session want us to just learn how um what are the different capabilities or I'm not sure if it's going too far fast for me I'm definitely feeling it's too fast yeah so what we're going to do in uh so we're going to use Azure remote for rapid prototyping of models at the boot camp okay okay so I want you guys to get more or less used to the interface and basically the modules that you'll be using at the boot camp right because we'll be building models in like 15 minutes to two hours uh per model all right so the idea is I I don't want you guys to get confused uh because of the interface or anything like that okay okay so the idea is I want you to get used to the interface like all right now I need to drop a column how do I do that all right I drag in this module now that I need to clean this data what do I do all right I'll drag in this module okay so it has a lot of capabilities and so it's of course it's going to be uh a little bit of a learning curve right but not as much of a learning curve as teaching you guys R okay so R is a full programming language so if if this is your first time going into machine learning uh with with uh without any knowledge of programming well now you have to juggle learning two things right you have to juggle learning programming you juggle syntax you're juggling the machine learning theory and you're trying to juggle the math as well and that's going to become really really hard so that's why we use Azure ml sometimes as a proxy right cuz sometimes we just want to split the data we don't care to multiply our indexes by 75 and then try to filter out all the indexes that weren't in that 75 right sometimes we want to just type in split and drag in this guy right and then this will split the data for us so the idea is this is going to be a drag and drop uh way to approach machine learning and from a teaching perspective this is amazing all right because you don't have to juggle syntax and programming and math at the same time you just have to juggle the the ideas behind machine Le the theory behind uh data data modeling and data machine learning okay so um we we'll have to be a little up to speed in terms of this um interface right I mean the Azure ml studio um and yeah I mean I think uh if if you could even suest resources in terms of what we could use before the um actual boot camp yeah get up to speed on this that will help so yeah so everything that we're going to use to get up to speed actually is in these webinars so everything I teach you today is completely relevant to what you're going to see uh uh at the boot camp okay same thing with the R so we'll give you homework for Azure ml also give you homework for our programming as well okay okay okay all right so going back into that all right so we we look at this data right so this data has some things with it so now we have to do what's called pre-processing right where we clean our data we conform our data we we make it uh we make it really nice and clean for our Models All right so we look at this uh so there's a there's a column here called symboling okay and we notice that it has only six unique values all right so that probably tells us that it's probably not a numeric feature and also you knows that it's all very clean integer-based numbers okay so that probably tells me that hey this can probably be a categorical value all right so what what does a categorical value mean so it's it's it's things that look like numbers but aren't numbers right so some good examples of that are let's let's see let's look at that right so categorical values right so would you say uh ZIP code is should be treated it as a number should we take the mean ZIP code should we whose mic is on oh there we go all right all right so uh if you look at ZIP code right should ZIP code be treated as a numeric value okay can we take the Min of zip code can we take the max of zip code can we average it all that good stuff we probably win right because a zip code is more or less a placeholder right it it it's a distinct value right it's what's called categorical value so a good good example uh of to test whether or not something's categorical is can we can we evenly put it into one bin or another bin right are you alive or are you dead right are you pregnant or are you not pregnant are you uh are is the stock going to go up is the stock going to go down all right so those are examples of of things that can be expressed in numbers but in this case probably shouldn't be treated as numbers they should be treated as distinct values or categorical virus okay so let's look at that so in this case it's going to be treated as it's right now it's treated as a numeric feature so what does that mean hey a quick question uh so are uh zip codes categorical values yeah zip codes are categorical values but uh they do not like one ZIP code to the next one like assuming 98 uh 052 and 98053 they have no relationship like uh I was thinking like you know when it comes to sizes for example small medium large those are categorical values yes so small medium large are categorical values as well however they are a separate type of categorical value they're what's called ordinal categorical value so ordinal means there is some kind of ordering to them there is uh you to get to one you have to get to the other like for example single married divorce right that is an example of a categorical value that has some kind of ordering you can't have one without without the other if you're divorced it has information in that in there because I know that you were single first and then I knew that you were married to result in the divorce does that make sense yeah and does the same apply to zip codes uh for the most part no uh it does in some cases some states but for the most part it's basically just incremented up by one it it's almost like a primary key in a database if you think about it that way so if whenever you need something new uh like a new record of something you would increment the database by one uh in the key and then that's your new zip code basically is how that is that that is formed sure thanks yep all right so this symboling thing probably should be treated as categorical value right so we want them to be treated as distinct and independent choices that could have been made okay does that make sense to everyone here all right so if we look at the next column it's normalized losses right so normalized what does normalized means it means we standardize the data right so we and then losses well let's take a look at this well this probably tells us this right here this distribution right here tells us that hey this can probably be treated as a number right and if we look at text right anytime we see text if we see text it probably means that it's either a string or a categorical value right it's there's no in between there there's no there's no chance of it being numeric all right so if it it if if it is a string now you have to ask yourself is it an actual string or a categorical value and how you can tell that is how many unique values does it have right so how many unique values if it doesn't have too many unique values then that probably tells you that hey this can probably be made categorical right so a good example of a categorical value that has a lot of different values is countries right there's like 300 countries out there there's 50 states right yes those are going to have a lot of unique values right but not enough so that they're all unique right it's not going to be in a situation where for example your name right right Ro Mohan and Sharon and things like yeah there's going to be a lot of duplicate Sharons but at the end of the day right if you combine your first middle and last name it's they're now now they're independent right they're they're they're kind of like pseudo keys for for for each person okay so in this case we should make make categorical as well we should make field type uh categorical because there's only two values here so there's two distinct values right Diesel and gas uh and also okay so we also look at this and there is hey there's two values in their aspiration right so number of doors okay so there's only two values and now look there's some missing value here right so a computer how a computer treats null is is it will break okay so if you ever tried to divide by zero it will break but zero is still there's still something there to divide by there there's still a number there but how do you divide by the absence of a number right so how do you add the absence of something right so the idea is you always need to have missing values filled in okay plugging in the holes if you will okay so we'll do that in a little bit as well if you also look at this uh there's also only five unique values so we'll make that category orical as well this should also be categorical there's three unique values two unique values 53 unique values so yeah so we're probably approaching the upper limits of what should be a unique value but if you look at this even further it has decimals right so usually categorical numbers have to be whole number integers right so in this case we should probably keep this guy numeric even though there's there's only 53 unique values yes it looks like there is some kind of scaling Happening Here uh with numbers so we should keep this numeric uh this is also decimal decimal decimal so we should keep these four guys decimals or numerics okay uh weight right so we look at weight there's a distribution here so yeah let's keep that a number so there's a string here and there's a string here and there's only seven unique values and seven unique values respectively so we can convert these guys to categorical as well okay so let's take let's keep track of which columns we want to make categorical okay so and let's get even further Okay so uh there is a uh data mining framework that we should go by right so once you get in data so get in data that should be your first step now what you want to do is you want to clean the data right so before that though you want to do what's called Data exploration but you also want to do what's called cleaning the data and there's many steps to cleaning the data okay so there is uh categorical casting okay so you want to the explicitly tell the computer that these values should be treated as distinct values or independent choices that could have been made okay and not treated as numbers another thing you want to do is uh filter out strings right so you want to drop strings because there's nothing the computer can do with that um or you can also transform the string so when we get to our text Processing Unit in in um in our boot camp we will talk about how do we take like things like a a Twitter message and turn them into something graphical that the computer can actually crunch okay and then we want to do things like clean missing data okay so let's look at that so right now what are we doing we're finding columns that we can be that can be made categorical okay so let's take let's keep track of that okay so in this case I have a column called symboling okay uh make so these should be made categorical okay fi type uh s expiration all right see number of doors there we go so number of doors and then there's body style uh Drive wheels so also you notice that when we're doing categor oral casting this this not this is not just an Azure ml this is in every machine learning platform so if you're data mining in Excel if you are data mining in in in anywhere else you will have to do these categorical casting so in R well you'll learn next week that you will convert these things to what's called factors so factors are how you explicitly call something a categorical value in rart okay so engine type okay uh number of cylinders so if we look at this too if number of cylinders was also written out explicitly in 1 2 3 4 5 6 right that can actually be made into numbers right so it's right now it's what's called an ordinal categorical value so we can actually just assign four the number four to the actual the digit four right so in that case there can be a range there right because if you have five cylinders I know that you have four 3 2 and one cylinders respectively okay all right so uh let's see engine type oh I already wrote that down all right so what else should we made categorical so there's also some kind of f system right and then Bor and stroke we can probably keep that as a numerics okay horsepower compression Peak RPM uh City MPG and all that stuff and right now we are trying to predict this guy all right so price will be our uh what we're trying to predict right and then every everything else will be our predictors okay so this is called the response class and these are called just predictors okay so uh based upon this entire data set can we guess based upon the the the statistics of the car can we guess the price of the car okay so at the boot camp you'll be learning how to do that all right so for purpose of this uh this webinar I'll just get as deep as how do we clean and process data okay okay so how do we make things categorical okay well let's look at this if I if I type in something called metadata editor and by the way you notice that there's lots and lots of tools in Azure ml right if I look through all of them I would probably go mad if I wanted to look for anything at all right so there's this nice search bar here okay so today I'm going to teach you something called the metadata editor and this is a really important uh module in Azure because it allows us to edit the schema or the metadata of our data okay so what does that mean so if I click on this uh on the right hand side there should be a toolbar so if you if you don't see it there should be this this this this bar right here and what this does is now we can change our our our data uh our data columns right so let's now we can launch What's called the column selectors so this is going to apply a transformation and now I want to ask and now it's asking what do you want to apply to transformations to well I want to apply the transformations to everything I consider to be a categorical value right so I have a I have a nice and handy list right here right so in this case if I click on this by the way so sometimes when you get into this it it it won't it won't autocomplete for you right away so if I just visualize okay if I visualize oh nope if I click on this and out and expand this toolbar there's a column selector if I launch it you notice that there's no autocomplete at first you have to click out of it and then click back into it and now there's Auto complete okay this is really handy right cuz now you can just point and click so uh I will make symboling make field type aspiration all right number of doors uh body style Drive wheel uh engine location engine type uh number of cylinders and Fu system all right these are the columns that I want to apply a transformation to okay so you notice that I could have done this many ways right so I I began with no columns so I want to begin with no columns and include only these columns okay so I can include them by column indices or column types or names as well right but if I check this now the whatever Transformations I'm make in this module will be applied to these columns and I want to go down here there's a categorical thing right here right so I want to make them into categorical value so if I run this now it's going to go ahead and transer my data and it's going to make my columns into categorical okay and if I look at this if I look at it before right it's see also there's another example here so number of doors that should probably be converted into two and four digits respectively rather than uh categorical values okay all right so if I if I look at the output of the metadata editor okay now we can if I Mouse over it now I can now see that Cate now symboling is now a categorical feature and you notice that there are now discrete values on the bar chart right on the bar chart now it's zero and one there is no there is no possibility of a 0 five symboling right uh this is this is important when you express your categorical features in terms of zeros and ones right so example of that is pregnant not pregnant right one representing pregnant zero representing not pregnant right so there is no such thing as maybe half prant okay you can't have .5 pregnancy okay it has to be one or a zero so if I look at the the the original uh data set when it was treated as numerics you notice that there is the idea of a 05 symboling there's the idea of negative point5 symboling as well okay so now that they've been conformed into categorical values you can see that visualizations start to appear now right so if I if I look on the if I click on any column A visualization appears so it's either going to be a box plot or a histogram okay so if I compare a categorical value to let's say make here so make is a normalized loss okay so that's a numeric feature by the way so a feature is just a just a the name of a column okay so each column is what's known as a feature in uh machine learning so if I click on symboling and I want to compare how symboling compares to uh uh normalized losses right we get box plots for each categorical value of symboling right just for example if I want to see the price of twodo cars versus four-door cars you can see that now I have a box plot and I can now log the box plot so I can see it much clearer all right so from this alone I can tell that for the most part twoo cars tend to be cheaper than four-door cars okay maybe it's because they're smaller I don't know okay so if you compare a numeric value to a categorical value which is what I just did right so price is numeric and number of doors is categorical right now so you see a box plot comparison of each categorical value for each Associated uh numerics okay but if I compare two numerics against each other so if I want to just see uh what is the what is the let's see the length of the car in relation to the price of the car so these are two numeric columns that I'm comparing to one another so if I go down here now we get a scatter plot so we see that there's some kind of exponential curve here right so as the gets longer it gets exponentially more expensive okay um if I compare two categorical values together like front wheeel drive and maybe that's a good what's a good one here so if I compare front uh wheel or the number the the the wheel drives to maybe the field type okay where is the field type I get a I'll get a grid okay so I get a grid that looks kind of like this and I wish they were expressed in percentages but hey we get uh color codings all right uh all right so that's that so now that everything is casted correctly now this is the part where we can do data visualization so if you're from a more traditional like kind of bi uh and and uh data analysis role right building reports and things like that right and you're building reports in R You're Building reports in in in in in Python right everything needs to be converted categorically first same with Azure ml okay now that we have that we have to in missing data right so how do we do that so there is something called uh uh the descriptive uh statistics module so I'll go to the top left hand corner and I'll just type in uh descriptive statistics I'll drag it in all right so now if I run this module it's going to give me what just summary statistics what's the first quartile range what's the second quartile range the median the max toin do the unique values and everything in the between but specifically I care about the number of missing values right so I could have gone through individually and and selected each column one at a time and see if it has missing values so this one has 44 missing values this one has zero this one has zero this but that's very tedious and gigantic data sets you don't really have the luxury if if it's a if it's a 1600 column data set you don't have the luxury of just doing that okay so there's a there's a cool uh module here that just summarizes what has missing value so I can see that normalized loss has a missing value uh bore and stroke and horsepower and Peak RPM and price also have uh missing values so let's I'm just going to write that down real quick so normalized losses uh number of doors okay and then I have a boore I have a stroke I have horsepower I have Peak RPM and then I also have price so price has missing values as well so that's also a problem okay because remember price is what we're trying to predict right so if what we're trying to predict is missing well basically the whole row is going to probably be lost so we'll look at that in a little bit all right so um in this case all right so let's explore these columns and see what they are so if I go back to my metadata editor so remember the metadata editor has my data set that's been transformed the descriptive statistics only outputs a table of the summary of Statistics okay so I can visualize the output of my metadata editor now okay so you can see that all right so we look at normalized losses so normalized losses is a numeric feature okay so I I should write that down right because how you treat uh missing values differs whether it's numeric categorical or string okay and we'll talk and we'll get into to that as as as these uh this goes on so let's also look for the number of doors so the number of doors is a categorical right so it's a categorical all right so what else we do we we also missing things in bore okay so bore has four missing values so this is numeric and stroke is also numeric I can just tell because it has decimals here horsepower that's probably numeric as well right because that's represented in in a number all right so Peak RPM that's also numeric okay and then we also have price which is a problem because it's our response class okay we're trying to predict price and we're missing values of price here okay so let's look at this so what do we want to do so in regards to clean missing data there is a clean missing data module so if I just type in clean drag in clean missing data all right so now I have a module that is just designed to scrub scrub data now I have to treat numeric scrubs separately from categorical scrubbing okay so let's talk about missing data for a minute missing data okay all right so when I'm cleansing missing data I have two I have two types of data right I have numerics and I have categorical and they're going to be treated separately and and differently as far as their Transformations are so numerics I have lot of Versatility versatility with right cuz I can fill in missing gaps in my data with things like the median the mean the mode the Min the Max uh I can uh of course I can drop the entire uh row I can drop the entire column I can build some kind of regression model right as some kind of statistical model so I can do things like Pearson's correlation or things like that or I can just build a another predictive model so I can build a predictive model that would predict for example uh based upon the vehicle's price and all the the good it's good stuff can I predict its normalized loss I can fill in the missing values that way as well okay but when it comes to categorical values I only really have a few choices right because you can't really do math uh on categorical values right because now that brings things out like the median the mean the Min the max but I still have the mode available to me right the mode is the most uh common type all right so if I look at this uh if I go back and it looks like number of doors was categorical right so it looks like the number of doors is categorical well the mode right now is four doors right so four doors is the most popular choice so in that case if I replaced it with the mode the missing data with mode anytime where we saw null oh can who's mic is that can we all right all right so anytime where you see uh a missing value for the number of doors it will be replaced with a four door okay does every understand that so if if if this column or this right here right this right here this is a this idea this is a null right here for the number of doors if we clean missing data with the mode now a four would now be introduced right here right and yes we are changing our data right and does anyone have a problem with replacing data with other parts of data you should cuz we're trying to predict based upon the basically the circumstances of of the car the data about the car can we predict the price but if we're altering the data by replacing it with the mode right now we have maybe fabricated data but let's talk about what happens if we don't fill it in with the mode if we don't fill it in with the mode well machine learning won't accept it in the first place okay so we should all agree that machine learning should still happen Okay because we want this predictive model to predict price okay so what are we giving up well we add in a four right here that wasn't there before and we don't know if this is the actual answer right cuz we're just filling it in so that the the model will accept uh the machine learning model will accept it so the mode what is the mode going to do well it's going to start overloading this column right here so four now it's now going to have two extra things on it that it didn't have before okay so now we're altering a little bit of our distribution all right so the histogram is going to change okay let's look at for example uh bore okay so if we replaced bore with something like the median right now our distribution is actually going to raise up around