You're NOT Managing Your Memory Properly | Python Generators (Yield)

here we have two different Loops that iterate over the exact same underlying data process it in the exact same way and have a similar run time well it might be easy to miss here is the size of each iterator in memory the first iterator is taking up 800 000 bytes while the second iterator is only taking up a residual 200 bytes in some cases the first iterator might even cause you to run out of memory as it scales up with the amount of data while the size of the second is independent let's dive into an example here and see what's actually happening let's say we want to iterate over some data where this data could essentially be anything and in this case we're just going to print it we don't actually have any data defined yet so let's go ahead and create it with a function called data that we're going to create now I'm going to give an arbitrary example at first just to illustrate the concept let's create a list to hold the data and then loop from one to n where n can be a parameter inside the loop let's go ahead and append a string that's going to be I characters long and then return this list when we're done all this function does is creates this pyramid looking list of strings of course we could have used a list comprehension instead but don't focus on that for now if we then Loop over this list with a size of 10 we can see our output if the arrays are input size to 100 we get what we might expect if we raise it to a thousand we get the same thing if we keep on raising it though let's say to 250 000 we run out of memory and a memory error is raised What's Happening Here is we're storing the entire list in memory even though we technically don't need to have access to each list element at the same time as We're looping we only need enough memory to store a single string at once during our Loop let's say we get to the case where we print out the string with five characters we only actually need enough memory to store a five character string next iteration we only need enough memory to store six character string so on and so forth the amount of memory the entire list is going to take up is dependent on its size with 10 strings it is 184 bytes with a thousand strings it grows to 8.8 000 bytes and with a hundred thousand strings it is 800 000 bytes you might be thinking that it seems inefficient to generate all over values at once when they are processed sequentially how can we design a solution where we only place in memory what needs to be processed at a given point in time without storing all of our data in memory together what if as we looped here we could generate a single value at a time on the fly right before we need to process it let's take a look at this function we know that when a function is called everything runs within its own scope this data variable is an example of a function scoped variable whenever we hit a return statement the function terminates all function scoped variables are destroyed and the resulting value is returned to the caller if the same function is called again sometime later the function will get a fresh set of new variables it turns out there's something we can do that prevents the local variables from being destroyed when we terminate a function this actually means that we can pause and resume the function halfway through this is known as a generator a generator is defined in the exact same way as a regular function however we need to include a yield statement instead of a return a yield statement will pass control back to the caller and the next time the function is called it will resume where the function left off with all of its variables still intact rather than building an entire list and returning it once it has been fully populated let's go ahead and insert a yield inside of our Loop and hand back just a single string at a time a generator is a type of iterator so all of the regular python iteration operations apply to it before I use it in our Loop let's go ahead and create an instance of the generator because a generator is a type of iterator you can't call it the same way that you call a regular function instead you must iterate through it for it to be called every time we perform a single iteration through the function it is called with its state left fully intact and we receive whatever value we wanted to hand back with the yield statement similar to our return if we create an instance of the generator and step through it with the next function it will execute until we hit a yield and return the value to the caller the generator now is essentially paused mid Loop and the next time you step through it it picks up where it left off if we go ahead and use this as our data in our Loop now we will get the exact same results however each string is generated right before it is needed the final results end up being indistinguishable the generator ends up using just a residual amount of memory as the code is not actually executed until it is being iterated over if you increase the size of n the amount of memory being used does not change as we run through our actual Loop now we will still be using memory but it will be just as much as we need for each string at a time rather than for all of the strings at once this is where the power of generators comes from as a quick digression it's important to note that the run times of the two different approaches will be nearly identical if we time how long it takes to run the data generation functions it looks like the generator function is far faster however this is to be expected as we didn't actually step through it yet if we time how long it takes to process each piece of data generated by each of the two functions we can see that the runtimes are nearly identical something else to note about generators or generator expressions in a similar way to how we could have written our regular loop as a list comprehension you can actually write a generator function as a generator expression a generator expression is going to function in the same way as a list comprehension however rather than having the logic within a set of square brackets we're going to place it inside a set of regular brackets we now no longer need a yield statement in this entire expression becomes a generator object not only do generators reduce memory usage and make it possible to compute across large data sets but they provide us with a multitude of other use cases for example they enable us to iterate through infinite sequences or real-time data due to being lazily evaluated the generator can essentially sit and wait for additional data to be available before yielding the next block of data to the caller to finish off let's go ahead and do the same thing but with a real world example within Finance rolling returns is a common analysis that can be performed on stock market returns it works by selecting a roll window and applying this role window to each consecutive period of returns across the sample the roll window needs to have the same granularity as the return series itself so if we have monthly returns we need to pick a role window that is some number of months in this case let's use a 36 month window we use the roll window to select the respective range of returns and then apply some operation to them in this case we're just going to calculate the annualized return for each window this means that we would select months 1 through 36 and then calculate its annualized return we then slide the window to months 2 through 37 and then calculate its annualized return then we move on to months 3 through 38 this continues until we have months n minus the window size to n or longer turns lends itself perfectly to generators as we have a lot of overlapping data and we only need to process a single rolling period at a time let's go ahead and implement this in code let's define a rolling Rats list that will hold our final results let's then Loop through each window within a list of periods and then append the annualized return of each period to the final results we can clean this up by implementing this as a list comprehension instead though to create a list of periods traditionally let's first create all of our Windows each window will start at a starting point and span the length of the Roll window each starting point will be a number starting from zero and ending at the sample size minus the roll window plus an offset of one we're then going to return each period's returns which will be the original sample split on each window accordingly this then gives us a list of 36 month periods the size of the list is almost 3 000 bytes however this is based off the size of the return series to turn this into a generator now we can simply switch our list comprehensions to generator expressions with little change to the structure of the code not only can we dynamically generate our periods but we can also generate our Windows as well since we only process a single window at each time step as well if we step through the generator we can see each subsequent 36 month return period the size of the generator is now only a residual 200 bytes of course it wouldn't be a rolling return analysis if you didn't plot our results at the end hopefully you now have a little more understanding behind the power of generators and can Implement them yourself in the appropriate cases if you made it this far thanks for watching I have a variety of topics that I want to dive into on this channel both in the form of informational videos and full length guides if there's anything in specific you want to see a video on feel free to let me know thanks for watching