Python Tutorial: Managing Data with Generators

Want to learn more? Take the full course at https://campus.datacamp.com/courses/parallel-programming-with-dask-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- We've seen that breaking files into chunks can alleviate problems caused by large datasets. Before introducing Dask, let's see how to use generators to help. Recall our work from before: we iterated over 1,000-line chunks of lines extracted from a CSV file of taxi-ride data. We then filtered the individual chunks applying a function within a list comprehension. If we replace the enclosing brackets with parentheses in a list comprehension, the result is a generator expression. Generator expressions resemble comprehensions, but use lazy evaluation. This means elements are generated one-at-a-time, so they are never in memory simultaneously This is extremely helpful when operating at the limits of available memory. We can quickly build another generator distances whose elements are totals of the trip_distance column from each chunk. No actual computation is done until we iterate over the chained generators explicitly (in this case, by applying the function sum to distances). To reiterate - no reading or work is done until the very last step. The generators chunks & distances persist after the computation. However, they have been consumed at this point. That is, trying to next function on either produces a StopIteration exception (which tells users that the generator is exhausted). Let's use generators now to read many files. We use a slightly different collection of CSV files that describe Yellow Cab rides in New York City. We use the string format() method and a generator filenames to generate names of CSV files for each month of 2015. Imagine here that, rather than having one large file to read in chunks, we have many large individual files that cannot fit in memory simultaneously. Let's examine one of the CSV files using `pd.read_c