PCA explained with intuition, a little math and code

Key Takeaways

hello everyone today we will explain how to get rid of too many dimensions especially after in the last video we have learned that too many dimensions can be often occurs so yes in a sense today we learn how to lift curses there are many methods for dimensionality reduction like pca ica bba funny yay okay i i messed up here again pca ica nmf lda ida disney umap autoencoders and so on and all are awesome coming with their own solutions and tricks but today we will discuss the basics by example of pca which stands for principal component analysis in a nutshell pca uses some heuristics to find the most important directions in the data then it can discard the most unimportant parts in order to achieve the desired dimensionality of the target space if the data would be two-dimensional we could even visualize it easily like this looking at the data clearly there are more important dimensions than others in this direction the spread is not that big and if we would clamp all points onto one line we would not lose that much information we would lose a lot of information if we would get rid of the other dimension of the spread because there the spread is much bigger and helps a lot to differentiate the points so the main idea of the pca algorithm is to keep the directions of the biggest spread and throw away the rest until we reach the desired number of dimensions d prime but how to do that with mass of course but first let's think about this intuitively what if we would approximate our data with an ellipse basically a two-dimensional gaussian and find out the main axis of the ellipse these axis which are called principal components are exactly the directions we are looking for we can keep the most important principal components and throw away the other ones but intuition does not compute stuff for us math does so let's see how we would do this mathematically by setting up the pca algorithm first let's suppose we have our data x which is a matrix with rows containing all our data points the columns contain the features of these data points then we want to centralize our data meaning that the mean of all points should be zero how to do that by taking the data and subtracting its mean done after this simple transformation we are ready to go on we want to find the biggest spread or variance of the data because this is the informative part the part most unpredictable of the data for that we compute the scatter matrix of the data defined by this formula and remember we have already subtracted the mean this is the reason why the formula for the scatter matrix is not more complicated but what is this thing the scatter matrix we remember that our data x is composed of rows of the dimensional vectors if we take one of these rows transpose it and do the multiplication we get a d by d matrix and of course we sum this all over our data samples i why have we done this we wanted to compute the main axis of the ellipse well it turns out that the directions of the main axis are actually the eigenvectors of this scatter matrix and each eigenvector has a so called eigenvalue which captures the importance of the eigenvectors so the magnitude of the spread so this is what we do we take the scatter matrix compute eigenvalues and eigenvectors for big matrices we do it of course in code like with this in python now that we have eigenvectors we can sort them by eigenvalue to determine their importance and we are ready to reduce the dimension for this we decide on how many dimensions we want to reduce say d prime then we take the prime eigenvectors and we put them into a matrix like this the eigenvectors are d dimensional since they live in the original space and what we want then is to compute the new position z in the new space of our data points like this meaning that we take out each data point x and multiply it with this matrix v of eigenvectors so we have x i one times the matrix which is a vector multiplied with a d times d prime matrix what comes out is a 1 by d prime matrix which is a vector so vectors in our new reduced d prime dimensional space so because v projects our points x from a higher dimensional space into a lower dimensional space d prime we call v the projection matrix what this projection matrix actually does is it rotates the data and maps it to a new space with lower dimensionality than the original one and that was it and the best part is we only have to understand the idea once because the implementation of the algorithm is already available in python with scikit-learn let's have a short look on how to use it this is one of the official scikit-learn examples where dimensionality reduction with pca is applied to the iris data set containing different flowers with four different features to differentiate between them like for example petal widths and links but we humans cannot really visualize the four dimensions uh until we do not reduce them to at least three dimensions and this is luckily exactly what this code does while the code might look dense we will now have a gentle introduction into it because you really have to know that pca relevant lines are very few the first python lines are as always importing specific packages like numpy for multi-dimensional arrays matplotlib piplot for plotting and of course sklearn for the machine learning part a scalern is so kind to provide us also with datasets so we load the iris data and save it into the variable iris in machine learning it is a custom to save the data into a variable called x and the labels in y we initialize the pca model by using the class from scikit learn as a parameter we specify the number of principal components we want to keep so the dimensionality of our target space d prime which is now 3. basically this is the line where we specify that our intention is to reduce from the original dimensionality of 4 to 3. and the cool part about scikit-learn is that you can exchange this line with many other dimensionality reduction algorithms but now with pca we fit the model to the data x and we do not use or need any labels y since they are not of any use in the pca algorithm which is a blind signal separation technique with the fitted model to our data which has internally done all complicated things for us like centralized the data computed the scatter matrix computed eigenvectors and sorted them after the eigenvalues formed the projection matrix with them so now it is ready to be applied on the data to compute the new representation which is stored here again in x x is now three dimensional and we are basically done we can use the rest of the code to make the 3d visualization that you see here at the side is it really that easy yes it is now you have no excuse not to use dimensionality reduction when needed do not forget to like and subscribe okay [Music] bye [Music] you