Python Tutorial: Data preparation for cluster analysis

Want to learn more? Take the full course at https://learn.datacamp.com/courses/cluster-analysis-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Hello everyone. Now that you are familiar with the two basic clustering techniques, let us discuss an important step in processing data that we should apply before performing clustering. Why do we need to prepare data for clustering? Imagine a situation where you have a set of variables with incomparable units - such as the dimensions of a product and its price. Even if variables have the same unit, they may be significantly different in terms of their scales and variances. For instance, the amount that one may spend on an inexpensive item like cereals is low as compared to traveling expenses. If we use data in this raw form, the results of clustering may be biased. The clusters formed may be dependent on one variable significantly more than the other. How do we account for these issues, then? We use a process called normalization. What is normalization of data? It is a process by which we rescale the values of a variable with respect to standard deviation of the data. The resultant standard deviation post normalization is 1. The process of normalization is simple and achieved through dividing a value by its standard deviation. Let us look at its implementation in Python. A normalization library is available on the SciPy package, using the whiten method of the vq class. First, we initiate the data to be normalized in a list and then use the whiten method, to transform. The array may be one or multi dimensional. Note that, in the example, we use a one dimensional list. In case of a multi dimensional array or list of lists, whiten divides each value by the standard deviation of the column. The result of the whiten method is an array of the same dimensions. Let us look at a plot using the matplotlib library in Python to compare the resul