t-SNE High-Dimensional Data Visualization | Python Tutorial

Want to learn more? Take the full course at https://learn.datacamp.com/courses/dimensionality-reduction-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- In this video, you'll learn to apply t-Distributed Stochastic Neighbor Embedding or t-SNE. While this may sound scary, it's just a powerful technique to visualize high dimensional data using feature extraction. t-SNE will maximize the distance in two-dimensional space between observations that are most different in a high-dimensional space. Because of this, observations that are similar will be close to one another and may become clustered. This is what happens when we apply t-SNE to the Iris dataset. We can see how the Setosa species forms a separate cluster, while the other two are closer together and therefore more similar. However, the Iris dataset only has 4 dimensions to start with, so let's try this on a more challenging dataset. Our ANSUR female body measurements dataset has 99 dimensions. Before we apply t-SNE we're going to remove all non-numeric columns from the dataset by passing a list with the unwanted column names to the pandas dataframe .drop() method. t-SNE does not work with non-numeric data as such. We could use a trick like one-hot encoding to get around this but we'll be using a different approach here. We'll create a TSNE() model with learning rate 50. While fitting to the dataset, t-SNE will try different configurations and evaluate these with an internal cost function. High learning rates will cause the algorithm to be more adventurous in the configurations it tries out while low learning rates will cause it to be conservative. Usually, learning rates fall in the 10 to 1000 range. Next, we'll fit and transform the TSNE model to our numeric dataset. This will project our high-dimensional dataset onto a NumPy array with two dimensions. We'll assign these two dimensions back to our original dataset naming the