PySpark Tutorial : Immutability and Lazy Processing

Want to learn more? Take the full course at https://learn.datacamp.com/courses/cleaning-data-with-pyspark at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- Welcome back! We've had a quick discussion about data cleaning, data types, and schemas. Let's move on to some further Spark concepts - Immutability and Lazy Processing. Normally in Python, and most other languages, variables are fully mutable. The values can be changed at any given time, assuming the scope of the variable is valid. While very flexible, this does present problems anytime there are multiple concurrent components trying to modify the same data. Most languages work around these issues using constructs like mutexes, semaphores, etc. This can add complexity, especially with non-trivial programs. Unlike typical Python variables, Spark Data Frames are immutable. While not strictly required, immutability is often a component of functional programming. We won't go into everything that implies here, but understand that Spark is designed to use immutable objects. Practically, this means Spark Data Frames are defined once and are not modifiable after initialization. If the variable name is reused, the original data is removed (assuming it's not in use elsewhere) and the variable name is reassigned to the new data. While this seems inefficient, it actually allows Spark to share data between all cluster components. It can do so without worry about concurrent data objects. This is a quick example of the immutability of data frames in Spark. It's OK if you don't understand the actual code, this example is more about the concepts of what happens. First, we create a data frame from a CSV file called voterdata.csv. This creates a new data frame definition and assigns it to the variable name voter_df. Once created, we want to do two further operations. The first is to create a full year column by using a 2-digit year present in the data set an