The curse of dimensionality. Or is it a blessing?

Key Takeaways

hello there in this video we will explain why many dimensions can be both a blessing and a curse when doing machine learning so it's miss coffee bean here with a video about some important basics in machine learning first we have to explain what are dimensions and why do we need them in machine learning the number of dimensions describes the number of coordinates we have to give in order to fully localize a point for example we have one dimension if we only need one coordinate to find the point like this point on this line for two dimensions we already need two numbers to localize one point three dimensions are very normal to us because space around us is three-dimensional and to localize an object we have to specify three points but the spatial dimensionality in particular does not limit dimensionality in general dimensions can be anything especially in machine learning problems for example to characterize a data point like a rocket we might need a dimension where we specify its speed we can specify the rocket's weight in a second dimension its height in a third dimension and so on in some machine learning contexts it is quite common to use the word dimension as a synonym for feature and theoretically nothing stops us to increase the number of dimensions or features to infinity we take whatever we can to characterize a point because these features can be then used by a machine learning algorithm to differentiate points one from another but what do high dimensional spaces bring with them a large number of dimensions can be a blessing think about problems that are not linearly separable in low dimensions so cases in which we cannot just draw a line to separate our data with for example linear regression but certain mappings to higher dimensions can make data linearly separable so very easy to solve however helpful in these cases many dimensions can quickly become a curse to visualize we have two points a and b in only one dimension to compute the distance between the two we just have to compute the difference between their coordinates done in two dimensions the distance computations become a little more difficult since we now have more coordinates for our points and if we assume that the points lie on the diagonal of a cube forming a 45 degree angle with both axes then the distance simplifies a little it becomes the difference between the coordinates in one of the dimensions say x multiplied by the square root of two and two happens to be the number of dimensions we observe the same pattern in one dimension where the difference of the coordinates is virtually multiplied by square root of one which is again our number of dimensions the same holds also for three dimensions where the distance between diagonal corner points of the cube scales with square root of three and here we generalize without delivering the proof now that the distance between such points a and b is proportional to the square root of the number of dimensions d what does this mean well it means that in a high number of dimensions there is a lot of space to begin with so we have a lot of new places to put points into since the distance between them is bigger with each added dimension when distances increase it also means that the overall volume of the space increases so that we need many more points to sample the space with the same density for one dimension it might be enough to sample only two points but in two dimensions one needs four points to fill a grid with the same interval and already eight points in three dimensions to be precise with each dimension we need exponentially more points to populate the space oh no what do we do high dimensional spaces are like a two-sided coin on the one side high dimensions are a blessing for linear separability on the other side of the coin there is a curse many dimensions increase the overall volume of the space so points and machine learning optimizers can get lost inside the void of high dimensionality so what do we do there are ways to decrease dimensionality and in the next video we will explain how dimensionality reduction works can't wait are you still here go and do not let the dark side of dimensionality take over see you bye [Music] you