R Tutorial : Introduction to Spark with sparklyr in R

Want to learn more? Take the full course at https://learn.datacamp.com/courses/introduction-to-spark-with-sparklyr-in-r at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Hi, I'm Richie. Welcome to the course! In this chapter you are going to learn how to work with Spark using the sparklyr's dplyr interface. Before we get to that, let's take a moment to explore what Spark is. R is a wonderful tool for data analysis, but by default the amount of data that you can process with it is limited to what you can store in RAM, on a single computer. For many datasets, that isn't a problem, but when you have really big data, you can run into trouble. Spark is a cluster computing platform. That means that your datasets and your computations can be spread across several machines, effectively removing the limit to the size of your datasets. All this happens automatically, so you don't need to worry about how your data is split up. Sparklyr is an R package that let's you access Spark from R. That means you get the power of R's easy to write syntax, and the power of Spark's unlimited data handling. The icing on the cake is that sparklyr uses dplyr syntax, so once you know dplyr, you are half way to knowing sparklyr. The one potential problem is that Spark is new, and sparklyr is even newer. That means that some features are missing, or tricky to use, and many error messages aren't as clear as they should be. This is the price you pay for being a trendsetter. You need to expect a little pain to gain all this power. The most important thing you will learn in this chapter is the workflow pattern. First, you connect to Spark, then you do your work, then you disconnect. Since connecting to Spark takes several seconds, it is sensible to connect once at the start of the day, and disconnect again at the end. dplyr provides a grammar of data transformation. There are five main transformations that you can apply to a datas