PySpark Tutorial : Machine Learning & Spark

Want to learn more? Take the full course at https://learn.datacamp.com/courses/machine-learning-with-apache-spark at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Hi! Welcome to the course on Machine Learning with Apache Spark, in which you will learn how to build Machine Learning models on large data sets using distributed computing techniques. Let's start with some fundamental concepts. Suppose you wanted to teach a computer how to make waffles. You could find a good recipe and then give the computer explicit instructions about ingredients and proportions. Alternatively, you could present the computer with a selection of different waffle recipes and let it figure out the ingredients and proportions for the best recipe. The second approach is how Machine Learning works: the computer literally learns from examples. Machine Learning problems are generally less esoteric than finding the perfect waffle recipe. The most common problems apply either Regression or Classification. A regression model learns to predict a number. For example, when making waffles, how much flour should be used for a particular amount of sugar? A classification model, on the other hand, predicts a discrete or categorical value. For example, is a recipe calling for a particular amount of sugar and salt more likely to be for waffles or cupcakes? The performance of a Machine Learning model depends on data. In general, more data is a good thing. If an algorithm is able to train on a larger set of data, then its ability to generalize to new data will inevitably improve. However, there are some practical constraints. If the data can fit entirely into RAM then the algorithm can operate efficiently. What happens when those data no longer fit into memory? The computer will start to use virtual memory and data will be paged back and forth between RAM and disk. Relative to RAM access, retrieving data from disk is slow. As the siz