Data Mining Explained | Data Engineering | Community Webinar

Key Takeaways

[Music] the purpose of this particular webinar is to give you all sort of some basic vocabulary and a very basic understanding of a number of different important topics regarding uh sort of data science fundamentals so a lot of this talk is a vocabulary lesson so it's really important that we that you guys make sure you understand all the terms that i'm introducing and all the ways that they're used we're going to be covering a lot of material over the next couple of hours so it is pretty aggressively paced uh but we should be able to get through all of it all right so you see on your screen here the topics that we're going to be covering so we're going to be talking to start about data and data types and sort of setting some ground groundwork uh for all the things we'll be talking about over the course of the bootcamp uh then we're gonna talk about data quality and data pre-processing which are very connected uh things and finally we're gonna talk about uh some similarity into similarity metrics and uh also some data exploration and visualization so uh we'll cover data exploration and visualization very briefly here uh we're gonna talk about it a lot more next week in the introduction to our webinar so without further ado then let's start with data and data types so what is data is a is sort of a very fundamental question that we can ask and here's where our vocabulary lessons start so data is a collection of objects that are defined by attributes so attributes are the properties or characteristics of our objects so every entry in our table here and not all data can be represented nicely in a table table but a lot of it can be so in this case the objects a data object is a row a data attribute is a column so we think of the attributes as being properties of the objects so the eye color of a person the temperature whether someone filed for a tax refund in the next year what their taxable income was those are all attributes of our data objects so one of the uh struggles people sometimes have in getting into data science is that because data science is in a lot of is a synthesis of you know probably three or four completely distinct fields all coming together in one way there are a lot of different terms for the same things in a lot of cases so this is our first encounter with that and it's going to show up again so attribute is sort of a decent name for these for these ideas but they're also called variables and fields and characteristics and features and predictors and if you've got tabular data they'll be called columns sometimes so all of those different names all refer to essentially the same to the same thing they're all attributes they're a property or characteristic of our object similarly when we have our objects so our objects are then basically a collection of attributes it's kind of a circular definition but it's with what we've got so each object is defined by its by its exact attribute values and objects we'll use the term data objects throughout this talk but in general objects have a lot of different names you'll see them called records and points and cases samples entities entries instances uh all of that and and many more sort of things you'll also see a set of data called a data set but sometimes it'll be called a table and sometimes you'll just hear oh yeah we have our data referring to the to the set as a whole so we have objects and we have attributes so each attribute has a set of values which the objects can draw from so each attribute each object is defined by an attribute by a set of attribute values and each attribute we can think of as being defined by the set of values that it can that it can hold so we can have the same attribute mapped different attribute values height can be measured in meters or feet temperature can be measured in celsius kelvin or fahrenheit lots of other sorts of things like that and different attributes will often be mapped to the same set of values id numbers and age are both usually given as integer values temperature and height are both often given as floating point values as decimal values so the properties of our attributes can also be different height for instance has a pretty practical maximum and minimum value as does something like age whereas id number has no real limit it's whatever we whatever the people who created the data set define it to be so and that kind of gets into this quote to a interesting question of who defines what value set that a given attribute uses and the answer that is essentially we do right the people who create the data set do people who hand us the data the data engineers or the street or the the twitter api that we're call our other apis that we're calling in order to get the data we'll have some definition of it but we can set that ourselves to we can change our attributes to have to be mapped to different sets of values and we'll use that in a variety of places all right so attributes have so we know that we have these attribute values so it's useful to talk about attributes as having being part of different class of classes different types of attributes that we're going to end up having to handle differently as we get into the actual data mining and modeling processes so there's two sort of fundamental types of attributes uh discrete attributes and continuous attributes so discrete attributes have either a finite or countably infinite set of values uh for those of you who don't know the for the term countably infinite basically means integers if you can turn your attribute into integers then it's con then it's countably infinite or finite if you've got only a limited set of integers so good examples of these are zip codes things like click counts the set of you know a word count in a word counts in a collection of documents right we could in theory have as many clicks as we want right there's a countably infinite set but they're always going to be integers so we have a countably infinite set of values there usually we represent these as integer variables um and binary attributes are a pretty special case of discrete attributes that we end up having to handle differently in some cases binary attributes have only two values and we might call those yes or no dead or alive one or zero and those kinds of columns are sort of a special case in some in some contexts we really like them they make things easier in other contexts they can be problematic which is pretty much everything the other big type of attribute classification that we that we see are continuous attributes so in this case we have real numbers as our attribute values there's no limitation to just integers so temperature height weight uh oxygen level taxable income all these things have um real numbers as their attribute values they can theoretically take any value at all now in practice of course we have to put these things into a computer and computers can only measure and represent a finite set of digits so generally speaking these attributes are usually represented as floating point variables so floating points for those of you who are farther out from your learning of programming are essentially just variables that hold a real number that can hold a decimal floating point being the decimal the floating point being the decimal point in the number all right so within these two sort of big categories of attributes we have some subsets that are also important to think about and one of the other one of the most important of these is the distinction between categorical attributes and non-categorical attributes so categorical attributes are discrete attributes that specifically have a finite set of values that they are allowed to take um so for instance uh so there's several examples here and within categorical there are two useful subsets so categorical values are any attribute categorical attributes or any attribute that have only a finite set of values if that finite set of values has a natural ordering so this is something like rankings or grades or clothing sizes um we call that an ordinal attribute so ordinal means that it has an order pretty pretty straightforward linguistics there um and we use and ordinal attributes are nice because we can code them as integers and maintain the ordering between them so we can we don't have to treat them particularly specially but most categorical variables are what we call nominal categorical variables or attributes so nominal attributes have no inherent ordering to them so eye color zip codes id numbers hair color whether someone is married or not or divorced or living with a partner there's no way you can say oh yes blue should have a value of five and green should have a value of two because i don't like green eyes right there's no there's no ordering that you can put into those variables um so nominal attributes in particular we have to handle uh we kind of have to be careful about handling um other useful types to think about in terms of uh things that allow us uh the variable types that allow us to to do to treat them specially in ways that are that are useful that are that are easier on the continuous side are interval and ratio variables you can certainly have intervals or ratios that are that are discrete but for the most part you see them as reals or as continuous interval variables are a variable where the measurement is of measurement basically where the difference between two values is constant and meaningful so for instance uh with temperature say temperature in celsius a temperature of 100 degrees and a temperature of 90 degrees have the same difference in heat between them as a heat of 80 degrees and a heat of 90 degrees so interval variables are basically continuous variables that have a nice metric we can assign them that gives us some nice handling uh something like the decibel scale uh on the other hand uh is much harder to handle as an interval because the decibel scale if you're thinking about the intent the actual intensity of the sound it's a logarithmic scale so the difference between three decibels and four decibels is smaller than the decimal the difference between 13 and 14 decibels so that's not that's an example of a continuous variable that isn't an interval variable all right we can move on to data set classification so data sets are there are a lot of different types of data sets and they require different approaches to analysis the pre-processing steps the modeling steps pretty much everything that you do with these different types of data sets is going to be different the kinds of models you use the kinds of visualizations you construct the kind of cleaning that is proper for that kind of data um understanding the structure of your data at the beginning is very important to not wasting time and not producing incorrect results and it's in this step the understanding the type the structure of your data that things like domain knowledge tend to be very important um but there are still certainly categories that tend to be similar no matter what domain they're in so uh we'll talk about these three different kinds of types of data sets records graphs and ordered data sets in a little bit of more detail coming up here so record data is data that consists of a collection of records each of which consists of a fixed set of attributes so this uh tax id so this particular data set which i use in some and some in several places is a record data we have every data object has one tax id has a value of whether they asked for a refund marital status uh whether they're single married or divorced a taxable income field and a ch and whether they cheated on their taxes or not so that's what sort of the structure of this data set so any data which consists of this kind of collection of records which consists of a fixed set of attributes you almost always represent this kind of data as a table um whether a database table or or a spreadsheet or something like that and it's the most common kind of data so a lot of people will if you talk about data or data sets this is what they visualize entirely as record data so it's sort of your your most common and sort of fundamental kind of data set so within record data there are a few useful subsets so this record data with the tax data has some categorical values and then one ordinal variable uh so tax id is ordinal right or is it it's really more of a of a nominal variable when you think about it because ordering doesn't necessarily matter right sure it takes numbers but 10 is not meaningfully different from five there's no ordering implied here so tax id is a nominal field a nominal categorical field uh tax refund is a categorical field marital status also taxable income is a continuous field so most data that you encounter has mixed data types like this you have some categorical some numeric and that's sort of your traditional type of record data if on the other hand your record data consists entirely of numeric attributes so this is entirely continuous entirely interval or ratio variables then we can think of it as a mathematical matrix rather than just a table so we would have an m by n matrix there are m rows one for each data object n columns one for each attribute and this is nice because we can think of these data objects as points in a multi-dimensional space where each attribute is represented along one dimension and that allows us to use a number of numeric techniques specifically involving distance that some algorithms not only make the make some algorithms easier but which some algorithms require there's a number of algorithms that require you to have data matrix data all numeric data so another useful uh sort of subcategory of record data is document data so in this case it kind of is is somewhat similar to data to a data matrix every term uh every entry every data attribute is a has a numeric value but in this case we've got counts we've got discrete values so in this case what we have here is each row each data object is represented by what we think of as what we call a term vector so this term vector in this case and there's several ways you can do it but in this case it just counts the number of times a given word appears in the document so document one has team appear three times play appear five but coach appear none uh document two on the other hand has coach appear seven times but never has play up here in the over the course of the document so because it is because these attributes are all discrete because they're all integer attributes we can do different kinds of things different kinds of algorithms and processing methods are more are appropriate than data matrices or mixed data is all right so the last special kind of record data that we're going to talk about here is transaction data so this shares some similarities to document data and you can uh do use some of the same analysis but there's different semantics around it as well so transaction data is exactly what it sounds like it's record data where each record involves a set of items so if we're at a grocery store the set of products purchased by a customer during one shopping trip constitutes a transaction and the individual products that were purchased are the items so we can the difference between this and document data is that usually these items have more information than just a count associated with them so not only is it bred there's a price associated with that there's maybe an inventory stock associated with that how many are left um all of those sorts of things so we can do sort of things similar to document analysis but there's other sorts of information we have to consider as well so that's transaction data so the next big category of data that we'll talk about briefly here is graph data so graph data the classic example of course is html is the is the world wide web is graph data is defined by as a graph it's defined by nodes which are our vertices in our graph so every web page is a node and then an ed a set of edges which point from one node to another and those edges can be one directional like here or they can be bi-directional here and then in addition to edges and uh nodes edges in some graphs have weight so in this case this count for if it's an html website this might be a count of the number of times that website this website here links to this website here so it links five times here but only two times here so when we're dealing with graph data and we won't talk about this in great detail because it's sort of its own sub problem that we don't have a lot of time to cover but it's good to be aware of um when you're dealing with graph data you have to put a lot of thought into how you capture the relationships between the nodes how you encode your edges and and vertices um we have to sort of you don't get the same kind of neat you know and there are n attributes that represent that can be represented by n columns right each vertices can have any number anywhere from zero to an into you know an infinite theoretical number of edges coming out of it so when you're analyzing doing that sort of analysis you have to handle it differently the last big category of data is ordered data now ordered data is data which has some sort of which where each data object has to be ordered in some way so in the case of a genomic sequence for instance um the ordering of our uh of our ribosome of our nucleic acids here gg ttcc etc is important right gg the fact that we have gg ttcc here is different than if we had had cc tt and then gg those are different these are fundamentally different sequences so we have to encode it in some way that preserves that ordering another example and sort of your classic example of ordered data is spatial and temporal data so this uh little gif here represents the average monthly temperature of land of both lands and oceans over the course of a year so in this case the the spatial aspect of the data is important where we are in the world certainly matters when we're looking at a data object and in this case if we were getting this data every row in say a database table might be might have a location associated with it and a time and there's an implicit ordering there both especially to the time but also the location so when we're handling ordered data we have to be very careful about it and this is very important because time series of course uh anytime you're thinking about doing any kind of sensing any kind of sensing material or anything like that you get time series data it's the most common type of ordered data and we'll talk during the boot camp a lot about uh during the back half of the boot camp especially about how we handle time series data now we've got sort of that basic definition those basic sort of we understand what attributes are and data objects and the different types of them we can move on to talking about data quality now data quality is particularly by new data scientists one of the most commonly overlooked or shortened or uh you know poorly shortened steps uh pieces of it get ignored get skipped because it just doesn't seem that necessary but understanding your data quality problems and understanding where they could come from is very very important to creating robust models that will actually work in production you have to know what you have to know what to expect in order to handle it appropriately so there are three fundamental questions around data quality right we have to ask this of every data set we get one what problems do we have to worry about how do we detect those problems and what can we do about those problems those are sort of the three fundamental questions you should ask yourself every time upon approaching a new data set and your early exploration should really be some of your earliest exploration should really be focused at answering these questions so i'm going to give you some examples of how we answer each of these three questions and some of the categories of things coming up so there are three very common kinds of data quality problems noise and outliers missing values and duplicate data these show up in production all the time so let's go through and sort of think about these in this context so those of you who have a scientific or signal processing background are probably familiar with the term noise noise in the data science context is when we have an invalid signal of some sort that overlaps valid data this obscures our actual attribute values and fundamentally what it means is that some of our data objects have invalid values in some of the attributes they don't have real um the inaccurate values there so examples of this in real life we have the distortion of a person's voice over the phone snow on old television screens pretty old crt television screens um noise can appear because of human inconsistency and labeling you see this a lot in sports for instance that require human judging there's a lot of inconsistency in how people get labeled there and just in general if you're trying to say rank websites for instance um human inconsistency and laser in labeling can be a real problem so as sort of a practical example of what noise can do when there's a lot of it this is a pretty straightforward signal we've got two sine waves here with different uh with different uh frequencies but the same amplitude there's a blue one and a green one um and if we in and so we can generate the sine wave it looks very clean very pretty we can even sort of distinguish the two different sine waves if we add those two waves together and then throw noise at it just sort of basic white noise like you might see in any kind of randomization thing and you end up with something that looks like this so the noise has completely obscured our actual signal um so noise is again fundamentally invalid data points that are that are obscuring our signals we have to be there's always some noise in any system it's just the nature of the universe sadly but understanding where your noise is at its worst and how you can deal with it is very important but even recognizing that it's there is the first step recognizing which of your attributes are noisy versus which are not are more noisy which is versus which of them are less noisy sort of the the complementary problem complementary problem to noise is the problem of outliers so outliers often look like noise at first their data objects that have characteristics that are considerably different from most of the other objects in the data set so if we look at sort of the the visual here we've got some sort of two-dimensional graphing of our data and most of our each dot each pixel point represents a data object that's been plotted on on a graph so we've got you know four clusters very kind of nicely defined clusters and then we've got these three other points just kind of hanging out in the middle of nowhere far away from all of the other data so the big distinction between outliers are that between outliers and noise is that outliers are actually valid values the data was collected properly it's clean but it's outside of the normal range the data object for some reason doesn't look like a normal object all right so so that's outliers and noise those are sort of the first category of data quality problems that get encountered a lot another one that shows up all the very frequently is missing values so sometimes missing values are because information is not collected so whether you're looking at census information or survey information in particular people will often decline to give their age and weight or will decline to give their annual income so you just have missing values other times the attributes that you're collecting may not be applicable to all cases right if you're asking people about the annual income of each member of their household on a survey well the children in the household don't have an annual income right it doesn't make sense so you just code that as a missing value and we'll talk a lot more about handling missing values when we talk when we get to data pre-processing but the fundamental ways we kind of the fundamental ways we can handle it are it throw out all the data objects that um throughout all the data objects that have any missing values we can estimate our missing values using means or medians or something else we can with some algorithms but not all ignore the missing values on a row by row basis or we can just throw the attribute out entirely which is something we might want to do if you know if we have an attribute that is 80 missing we probably just want to throw that column out and one of the ways one of the other things you can do sometimes in some algorithms is you replace missing values adaptively with uh this happens a lot in categorical where you'll count the probabilities of an attribute appearing attribute value appearing over your whole data set and then replace all the missing values with such that those probabilities don't change um and we'll talk a little bit more about that when we get to pre-process i guess where to sort of get the basic sort of this is how you handle missing values in a very basic sense out there um and along the third category then alongside missing values noise and outliers is duplicate data so this is particularly a problem when data objects are when we're merging data from heterogeneous sources so if we have some data from google analytics coming from our website and we have some other data from um from uh you know actual uses you know click counts and and and sort of dwelling time and things like that that's from another system or maybe we have a java applet that you know as much as those things still exist uh on the internet um that collects some data inside of it if we want to merge that data we will sometimes have duplicate data objects we'll have the same person with multiple email addresses we'll have the same person represented with two different ids because they're coming from two different systems so generally speaking duplicate data though is pretty easy to handle uh assuming that you can detect it properly which is get rid of the duplicates merge it together but if you've got data that's heterogeneous that's from headers from multiple sources um then you do have to be really careful about about filtering out your duplicates so now we get to the much uh foreshadowed data pre-processing section so data pre-processing is sometimes called data cleaning but data pre-processing should should involve more steps than just cleaning the data just removing the problems with the data so data cleaning is kind of a subset of pre-processing but most of what we do during data pre-processing is in fact data cleaning so you'll again lots of different terms to refer to basically the same thing so there's a lot of different types of pre-processing uh and i'm gonna talk about a lot of different strategies aggregation sampling all the ones on the screen here i'm gonna talk about all these different strategies but um we don't want to use all of these different strategies on every data set right we want to there's a lot of different strategies we can use but for any given data set we're only going to use a couple of them usually we don't want to overwhelm we're not going to need every technique in our every tool in our toolbox every time another note before we keep going not all of these are strictly independent they all get these terms categories are all things you get you see thrown around in terms you see used around the the industry but because again data science is such a heterogeneous field um it's not not all of these things are strictly independent so if you see some overlap in what i'm talking about between different attributes that's why so first strategy and this one is when we because first because we see it a lot is aggregation so we'll combine two or more attributes or objects into a single attribute or object so this can be where we are trying to reduce the scale of our data trying to reduce the number of attributes or objects so we could for instance combine two attributes to combine a high temperature attribute and a low temperature attribute in order to get a temperature difference attribute we've now come we've now combined two columns into one column all of the basically every algorithm has some time dependence on the number of attributes it runs and certainly in terms of visualization and exploration there's only sort of so many attributes that you can hold you can look at at the same time or hold in your head at the same time on the other hand we might want to combine a bunch of different objects if we have users who have many different sessions or who navigate to many different pages we'll have dwell times that are that are that are different for every page and every session and we might want to combine average all those dwell times in order to get one data object that is sort of the average user behavior for each user rather than the you know 10 or 15 different sessions for that user um so the reason why we do this is is exactly that we want if we want to average user times for instance we're changing our scale we want to aggregate cities into region states or countries we want to aggregate dwell times across sessions or across pages and one of the big advantages of aggregation particularly averaging is that aggregated data tends to have less variability it's a way of reducing the effect of noise well it's a way of reducing the effect of random noise if you've got human labeling errors then you've got human labeling errors if you've got uh you know if you've got sampling procedure errors you have sampling procedure errors but if you've got random errors say you're you know random noise then aggregated data will very much tend to reduce that so as an example of that and i really like this next page for this these two are graphs of precipitation in australia so every so these are histograms so the uh height of each block uh is the number of locations where um where precipitation was measured which had in this case a standard deviation of between zero of the x value when we measured it on an on a monthly basis so we're measuring the average monthly precipitation and measuring the standard deviation of that monthly precipitation at you know 500 different land locations in australia when we do that on a monthly basis we get this very wide spread of standard deviations some places are very consistent in their rainfall other you know there's kind of these two peaks and then you have this long tail of places that are just all over the place in terms of the variability in precipitation on the other hand if we take those exact same land locations and instead find the average yearly precipitation the very variance standard deviation of that we get this very nice single peaked mostly single peaked very short-tailed histogram we've significantly reduced our variability we've reduced our random noise in our data set by uh by increasing the scale by aggregating our data over a longer time period so that's one of the big reasons that we use aggregation another very common method of pre-processing is sampling so those of you like ron who are from a statistics background um will know will understand sampling quite well so sampling is the main technique that we use for data selection it's used almost always for preliminary investigation of the data but it's often used even for the final data analysis even in data science statisticians have been sampling for the duration for the length of time that their discipline has existed because obtaining the entire set of date of interest is either too expensive too time-consuming or even in a lot of cases theoretically impossible there is no way that you can sample that you can obtain the entire set of some kinds of data it's just not possible so you have to sample carefully data miners sample often because processing our entire set of data is too expensive or time consuming if you talk about someone like a group something like linkedin or facebook or google you're talking about hundreds of terabytes into petabytes worth of data that they have stored in their servers it's in it just would take you cannot process that kind of data on anything remotely resembling a human lifespan even with modern technology we can process a lot of data but there's still a fundamental limit of what we can process and on top of that there's a fundamental limit of what we as humans can look at what we can really get what we can look at all at the same time so when you're sampling there is one thing more than anything else that you have to keep in mind which is representation so the key principle when you're sampling is that the sample will work almost as well as using the entire data set if and only if the sample is representative so and representative is sort of one of those fun words that means something different for every data set right so sometimes representative is as easy as unweighted random sampling other times this is particularly true if we're doing something like anomaly detection we need to make sure that whatever sample we take has an appropriate proportion of anomalies versus normal data in other contexts it gets even more complicated sometimes we want to make sure we balance out our different classes in a classification context or that certain kinds of attribute values that aren't even that aren't even target values but attribute values are all represented in a certainly way and balachander notes that sampling will typically exclude outliers and may have noise and that's absolutely true sampling if done improperly can absolutely add noise to your data um or well not really add noise in our context but certainly um but certainly can introduce noise and outliers are probably not going to appear because you don't sample enough to make them appear and that's true and that's actually one of the advantages of them of sampling is that it will exclude outliers most of the time so if we aren't in an anomaly detection context and we don't care and we kind of don't want outliers muddying the waters so to speak um we'll want to exclude them and sampling can help us do that there are several different types of sampling um that are important uh that sort of will come up as we talk about the over the course of the boot camp so there's simple random sampling where there's an equal probability of selecting any particular item um there's stratified sampling where we split the data into several partitions and draw out random samples from each partition if we're doing stratified sampling with equal sized partitions then that's equivalent to rant to simple random sampling but in a lot of cases we don't do it with equal sized partitions we do it with smaller or larger we have different sized partitions to draw from which is what makes it fundamentally different from simple random sampling or we are drawing different numbers of points out of out of the different partitions um so those are sort of two different sort of our two fundamental ways of of actually of of grouping the data and then when we're actually sampling there's two kinds of sampling that come up the sampling without replacement which is what most people think of when uh they're thinking of sampling uh so sampling without replacement is if we have a bag and it's got five red balls and four blue balls and three green balls in it and we reach into the bag and pull a ball out we say aha i drew a red ball then we take that red ball and we put it on the table and then if we want another item we reach back in and pull out a different ball so now the second time we draw instead of there being five reds and four blues and three greens there's four reds four blues and three greens so in so that's the sampling without replacement we do not replace what we're sampling back into the bag on the other top hand there are uses in this actually one of the most important um a fundamental concept uh a very common type of of modeling uses sampling with replacement as part of it so in sampling with replacement instead of taking the red ball out and then putting it on the table and drawing again we reach into the bag draw pull out a ball say aha it's red note down on a piece of paper say that it's red then put the red ball back in shake it up and draw another ball out again record its color put it back in the bag so without replacement with replacement that's exactly what it sounds like but they end up having very different uh mathematical results and as a result and because of that they are used in different contexts alright so the last thing we need to think of another another aspect we need to think about around sampling is what size of sample we want to do and i really like this picture because i think that it very excellently illustrates the problems with sample sizes so when we sample we do lose information just like with aggregation so you have to be um careful not to make your sample too small so if we look over here we have this data set and it's just you know position data this is i think some sort of lithography picture so we've got these black structures and then we've got this sine wave in the background and then a little bit of just sort of random noise scattered all over the place so if we subsample this by a quarter so we sample 2 000 points we can still see the structures the big thick structures are still represented but the sine wave has almost entirely disappeared we've lost that background image and if we go down even farther if we subsample by another quarter down to 500 points we've lost even the information of of these things like you can look at this and you can kind of see the structures but only because you know what the structures need to look like if i showed you just this graph first you wouldn't pick out the structures you wouldn't be able to there's just not enough information there so we want to reduce our sample size we want to sample a small enough size that we can process it efficiently that we can analyze it efficiently that we can explore it efficiently but we have to be really careful not to take too small a sample and unfortunately there really isn't a good rule of thumb on this necessarily um but you just need to you need to play with it you need to you know take lots of different samples of different sizes you need to do this to figure out when your information starts to disappear the next kind of thing we're going to talk about is what's called the curse of dimensionality so this is as much a data this is sort of a data quality issue but it's something that we have to be careful about when we're doing data pre-processing so the curse of dimensionality is that as your number of dimensions increases so as the number of columns number of attributes you have in your data set increases the data inherently becomes increasingly sparse in that space since in a lot of contexts in a lot for a lot of different algorithms definitions of density and distances between points of similarity and dissimilarity are really important to things like clustering methods and outlier detection so anomaly detection and this all becomes less meaningful if you add enough dimensions every point looks like an outlier so a great illustration of this is that if we randomly generate 500 points in a in a in an n-dimensional space and we compute the difference between the maximum distance between any pair of points and the minimum distance between any pair of points and this has been normalized in a log taken to make it look pretty we can see that in at two dimensions with 500 randomly generated points the maximum distance is about three and a quarter times larger than the minimum distance actually this is ten to the three and a quarter times larger because there's a lot there's a log base 10 here as we increase the number of dimensions though that spacing falls off really sharply and by the time we get down here 30 40 50 dimensions our points are so sparse that the minimum distance between points and the maximum distance is almost the same thing this is a this is this represe this 50 point represents a factor of something like 10 to the 20 10 to the 0.25 like the fourth root of 10 is the difference between the maximum distance and the minimum distance is just a very small number it's really hard to define outliers when you have such high dimensional data because every point is an outlier on in in some ways because there's just so many there's there's just so the space is so sparse so the solution to this data quality problem is something called dimensionality reduction so we can do dimensionality reduction via aggregation um or other sorts of of column combination um but there are also a number of uh mathematical techniques two of the big popular ones are principal component analysis or pca and singular value decomposition also called svd um and those are mathematical techniques that will run automatically that will reduce the dimension mentionality of your data pca actually usually goes from n dimensions so as many dimensions as you happen to have all the way down to two dimensions uh natalie they are kind of the same thing but they aren't exactly the same thing i'm not gonna go into great detail because we don't spend a lot of time on dimensionality reduction over the course of the boot camp but my understanding is that they are distinct um techniques though they have the same goal they just are different they have the same goal but they are achieved via different mathematical methods all right so another way to reduce dimensionality to reduce dimensionality of data other than just [Music] pca is a lot of times we have redundant or irrelevant features so this is going back to teresa's questions about dimensions being independent so a lot of times we have uh so if we have redundant features or relevant features that will increase our dimensionality artificially it won't it'll be it has it contains little to no information but it increases our dimensionality so we want to be very careful about trying to detect things so a redundant feature example for instance is that the purchase price of a product and the amount of sales tax paid on that product those things are you know based on the state completely connected you can calculate one from the other they're perfectly correlated so as a result you want to get rid of it because it increases your dimensionality without adding new information same thing with irrelevant features uh a student's id number the vast majority of the time is irrelevant to the task of predicting students gpa and it isn't just these types of redundant and irrelevant features don't just harm us via increased dimensionality redundant features effectively weight features multiple times if we have the same information contained in two columns two separate columns that the model thinks aren't that model thinks are both important we've double weighted that information similarly irrelevant features can confuse our model it'll the model will try to do some fitting based on those features and it'll just sort of diffuse the effectiveness of the model so we need to be so one of the one of our big steps of data pre-processing is making sure we figure out what attributes are redundant and irrelevant and aggressively cutting them out of our data set and there's a number of different a number of different techniques you can use to do this kind of subset selection you can brute force it just try all your different preacher subsets some algorithms some of the most popular algorithms used actually naturally do feature selection and so that's that's always good [Music] sometimes you have a filter approach where you use your exploration and what you know about the data set in order to filter out the bad features um and sometimes uh you can get the uh the data science inception going on where you use a data mining algorithm on your data mining algorithm in order to find the best subset of attributes uh but that's feature subset selection it doesn't share a lot um i'm gonna move on a little quickly please ask questions as they as they arise to you but we're running a little bit behind which is great i love the discussions we've had and it's important the front half of this presentation is more critical in the back half but i am going to start increasing the pace a little bit just as a heads up so please ask your questions as they come up so another uh common technique uh and this kind of goes with aggregation to a to a certain extent um is feature creation so sometimes we don't we have the cursive dimensionality on the one hand but other times we don't have enough features we don't have enough information there's more information that we could have so we can either extract things say combine two columns that in order to get new information so uh for instance in sales we could determine the tag price from the total amount paid kind of filtering out the um the the the sales tax which might be important other times we have aggregation and things like that with feature construction um and last and really mostly least because we don't do this that much is mapping data to a new space so those of you from a uh from a scientific background are probably familiar with the fourier transform um which uh takes data that is in the time domain and converts it to be in the frequency domain um which allows you to pick out different pieces of information we don't do this kind of transformation that much in data science because it tends to require transforming the entire data object but it is something to be aware of to have in your back in the back of your head because there are some times uh that you really do want to do some sort of massive transformation like this particularly in an anomaly detection time series context you're going to want to you might want to do things like take a fourier transform of your data and in addition to doing something uh very complicated like a fourier transform you can take a lot more similar a lot more straightforward transformations of your data so very common transformations are taking the exponential of the data taking the logarithmic the logarithm of a data value um taking the absolute value of a data value um all of these allow us uh to all of these uh types of things allow us to very nicely um to try to bring out different dependencies in our data to try to correlate our data our data attributes better with whatever our target is the two the other two things here i'm going to take special time to talk about because they show up a lot so standardization and normalization are probably the most common kinds of transformations that are applied to data to attributes in data science standardization is where we take our numeric data and we divide the numeric data each numeric uh value by the mean uh we subtract sorry we subtract the mean and divide by the standard deviation of our data set so what this does is it forces our data to have a mean of zero and a standard deviation of one so that's why it's standardization um what this the reason why we do this is that a lot of times is that it's a way of scaling our data down if you have for instance age and uh annual income there are a lot of different really the majority of uh model of algorithms won't will overweight your data science or will will overweight your uh annual incomes so if you have agent annual income so but if we standardize both of those then age and annual income are going to be weighted in exactly the same way um a somewhat less extreme version of of to do the same thing is normalization where we simply subtract the minimum from in it from every data value and then divide by the maximum and that maps the entire data onto the range from zero to one um it distorts the uh separation between the values to a certain extent um but it it does scale it very nicely so that age again taking the age versus annual income distinction age and annual income will end up on the same on the same zero to one scale they'll be weighted the same way by our by our algorithms so the next uh section is similarity and dissimilarity i'm going to kind of blast through similarity and dissimilarity uh to get data exploration and visualization um if we have to cut data exploration and visualization a little short we will because we're gonna talk about it a lot on uh i think it's tuesday when we do um when we do the on monday when we do the introduction to r uh when we do the introduction to our lab webinar so similarity and dissimilarity are exactly what they sound like similarity is a numerical measure of how alike two data objects are uh it is higher when more when objects are more alike and you usually set it up so that it falls in the range between zero and one dissimilarity is a numerical measure of how different two data objects are it's lower when objects are more alike the minimum dissimilarity is zero almost always the upper limit varies based on the exact metric you're using but is often one just sometimes higher um and we'll use the term proximity to refer to either how similar or how dissimilar objects are so so objects that are close will have a high similarity or a low dissimilarity and if objects are far if they have a low similarity and a high dissimilarity now in the context of data matrices of very nice numeric data we can use something very straightforward we can use simple distance formulas but in the context of ordinal of categorical data numeric numer uh uh nominal or ordinal data then we often need to use different things so nominal data we usually use the sort of binary similarity dissimilarity measurement where uh dissimilarity for nominal is dissimilarity is zero if the two values are the same and one if the two values a