Dimensionality Reduction | Introduction to Data Mining | Part 13

Key Takeaways

the next kind of thing we're going to talk about is What's called the curse of dimensionality so this is as much a data this is sort of a data quality issue but it's something that we have to be careful about when we're doing data PR preprocessing so the curse of dimensionality is that as your number of Dimensions increases so as the number of columns number of attributes you have in your data set increases the data in apparently becomes increasingly sparse in that space since in a lot of contexts in a lot for a lot of different algorithms definitions of density and distances between points of similarity and dissimilarity um are really important to things like clustering methods and outlier detection so anomaly detection and this all becomes less meaningful if you add enough Dimensions every Point looks like an outlier so a great illustration of this is that if we randomly generate 500 points in a in a uh in an N dimensional space and we compute the difference between the maximum distance between any pair of points and the minimum distance between any pair of points and this has been normalized in a log taken to make it look pretty we can see that in at two Dimensions with 500 randomly generated points the maximum distance is about three 3 and a/4 time larger than the minimum distance actually this is 10 the 3 and a/4 time larger because there's a lot there's a log base 10 here as we increase the number of Dimensions though that spacing falls off really sharply and by the time we get down here 30 40 50 Dimensions our points are so sparse that the minimum distance between points and the maximum distance is almost the same thing this is a this this repres this 50 point represents a factor of something like 10 to the 20 10 the 0.25 like the fourth root of 10 is the difference between the maximum distance and the minimum distance it's just a very small number it's really hard to Define outliers when you have such high dimensional data because every point is an outlier on in in some ways because there's just so many any there's just so the the space is so sparse so the solution to this data quality problem is something called dimensionality reduction so we can do dimensionality reduction via aggregation um or other sorts of of column combination um but there are also a number of uh mathematical techniques two of the big popular ones are principal component analysis or PCA and singular value decomposition also called SVD um and those are mathematical techniques that will run automatically that will reduce the dimension dimensionality of your data PCA actually usually goes from n Dimensions so as many dimensions as you happen to have all the way down to two Dimensions uh nalie they are kind of the same thing but they aren't exactly the same thing I'm not going to go into great detail because we don't spend a lot of time on dimensionality reduction over the course of the boot camp um but my understanding is that they are distinct um techniques though they have the same goal they just are different they they have the same goal but they are achieved via different mathematical methods