Python Tutorial: Delaying Computation with Dask

Want to learn more? Take the full course at https://campus.datacamp.com/courses/parallel-programming-with-dask-in-python at your own pace. More than a video, you'll learn hands-on coding & quickly apply skills to your daily work. --- We've introduced generators to defer computation & control memory use. Let's use Dask to simplify this process. We'll get to real data soon, but we'll start with simple function composition. We define three ordinary functions f, g, & h with the def keyword as usual. Each takes a single numerical input & returns a single numerical output. We then perform a sequence of computations, assigning intermediate computations to x, y, z, and, the final result, w. This is, of course, equivalent to nesting function calls without labeling intermediate results. We repeat this computation using delayed from the dask library. This is a higher-order function or a decorator function that maps an input function to another modified output function. The value w, then, is delayed of f of delayed of g of delayed of h of 4. If we examine w, it is a dask Delayed object rather than a numerical value. The delayed decorator stalls computation until the method compute() is invoked. The Dask Delayed object has another method visualize() that displays a task graph in some IPython shells. This linear graph shows the execution sequence & flow of data for this computation. Let's repeat our computation, this time reassigning the identifiers f, g, and h. The result is the same, but the functions f, g, & h are now decorated permanently by delayed. This means they always return Delayed objects that defer computation until the compute() method is called. To recap, we can define a function like f and rebind the label f to the new function obtained after applying the decorator delayed to the original f. The @ symbol here is an equivalent shorthand notation to decorate functions in this manner. Here, the @ symbol means "apply the decorator function delayed to the functio