Python Tutorial: Thinking about Data in Chunks

Want to learn more? Take the full course at https://campus.datacamp.com/courses/parallel-programming-with-dask-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- We've seen that available memory & storage restricts datasets that can be analyzed. A common strategy is to subdivide datasets into smaller parts. We'll use a 200,000-line file summarizing New York City cab rides from the first two weeks of 2013. Then, using read_csv() with the parameter chunksize=50000, the function returns an object we can iterate over. The loop variable `chunk` takes on the values of four DataFrames in succession, each having 50,000 lines except the last (because the first line in the file is the header line). The loop variable chunk has standard DataFrame attributes like shape. So the last chunk has almost 50,000 rows & 14 columns. Calling the info() method shows the column names like trip_time_in_secs & trip_distance. We can construct a logical Series is_long_trip that is True wherever the trip time exceeds 1200 seconds (or 20 minutes). Recall we can use the dot-loc[] accessor with the boolean Series is_long_trip to filter rows where this condition holds. The shape attribute reveals about 5,500 taxi-rides longer than 20 minutes in duration in this chunk of 50,000 trips. Let's embed this filtering logic within a function filter_is_long_trip that accepts a DataFrame as input & returns a DataFrame whose rows correspond to trips over 20 minutes. Next, we make a list of DataFrames called chunks by iterating over the output of read_csv, this time using chunks of 1,000 lines. Rather than initializing an empty list chunks and appending elements within a loop, we can also use a list comprehension to build the list. Remember, this list comprehension is equivalent to the preceding for loop. In both cases, each chunk is filtered as it is read from disk. We can use another list comprehension called lengths to see that t