When Should You Use Generators in Python?

Key Takeaways

what is going on guys welcome back in this video today we're going to briefly talk about when you should use generators in Python so let us get right into [Music] it all right so we're going to talk about generators in Python today and this video is going to mainly focus on the application and the use cases of generators when to use them and why to use them not so much on what they are and how they work I already have videos about this on my Channel as well you can watch those if you want to I am going to give you a very brief introduction and summary of what generators are in this video but I'm going to mainly focus on the application Fields so let us get started by just explaining very briefly what they are let's go to the coding directory here main.py and a generator is basically a function that generates values so I can define a function my generator and this function then generates values by using the yield keyword so I can either do it manually like this yield 10 and then maybe I can do yield 20 30 40 50 and this is then my generator function and I can Define it I can say gen is equal to my generator and then this generator function produces values how does it do that I can say next I can call next on the generator and it's going to give me the next value of that generator so if I run this you can see 10 20 20 30 40 50 because these are the values yielded by the generator if I try it again we're going to get uh we're going to get this stop iteration error because we don't have any more values and I can also iterate over generators I'm giving a very fast introduction now if you want a more detailed explanation watch my other videos but I can say for Value in generator print value this is also possible then it has the same behavior but the interesting thing about this is this is not a collection this is not a list where all the values are present already this is a function that generates values and it can do that either with individual yield statements or it can also do it uh like this for I in range 100 yield I this is also possible so that's basically it this is my introduction now into generators for this video the question is now why do we need that when should we actually use generators why should we actually use generators and I have prepared a couple of examples here uh which I've called memory F memory gen Fibonacci F and binary search tree and they're going to explain basically use cases for using generators now one major reason to use generators is memory efficiency or limited resources not having enough space not having enough memory which can be because you're working on a device it has very small amounts of memory or you're just working with massive files maybe you have like a huge file you have files that are way too large to load them into RAM maybe you can load them on the disk maybe it has like 100 gigabyt of data so you can store it on your disk but you cannot load it into RAM at once and generators basically allow you to go step by step through the lines for example through pixels whatever uh step by step through portions through chunks of the data rather than uh processing everything at once so to show you that I have here A Memory full py script and what the script does is it opens up a file that I downloaded words.txt this is just a long file I think we can take a look at it here LL if we look at the words file you can see here it has four what is this four megabytes of data not too large but imagine this to be a large file and what we can do now is we can either load the whole file into memory and then process it in this case uh we just load it re read the lines and that's it we can either do that or we can do it with a generator that goes line by line so what we have here is a setup where we have a memory usage function this just checks how much memory we're using how much memory this process is using um and then it loads all the lines and it terminates the script and just gives us how much memory was used in this process I have the same script here for a generator or by using a generator and this function here now opens the file and yields the individual lines it doesn't store them in a list it just yields them and what we do then is we iterate over them and we don't do anything with them we don't print them or anything but we process them in the same way we iterate over them but we don't store them in a collection we don't store them all at once so this is a very simple example but if I run these two files now if I run memory for. py you can see that this uses 32 megab of RAM and if I run memory gen it uses 0.12 megabytes of ram so way less because we don't have to store the full thing in the ram so you can think about this maybe let's go with my paint here I don't have my drawing tablet here right now but I can use my mouse maybe imagine you have like a huge file that you want to process what a list does or what loading all of the file does is it stores all of this file into RAM and then it processes this file what you do with a generator is you take a portion of that file into RAM you process it and then you say give me the next one and then you take that into RAM so you need very little Ram to process it like that you don't have all of the data in your memory at the same time so that is one major reason to use uh to use generators and also this lazy evaluation this I only get the value when I need it saves processing time uh CPU cycles and memory another reason to do that is to represent or to another reason to use generators is to represent infinite sequences and for this I have the Fibonacci sequence here because the Fibonacci sequence doesn't really have a limit so I have here if fa Bonacci list I can say I want to have a Fibonacci list up until a certain point uh with a function so I can generate the list and return it but actually the Fibonacci sequence doesn't have an end it's the rule of taking the last two numbers and adding them together to produce the next number this is an infinite sequence now you cannot have an infinite sequence as a function but you can have it as a generator because you can always get the next value by applying the rule and again here we have um the same idea we take into account time and memory usage if I run this here Python 3 Fibonacci full you're going to see hopefully this is not too much maybe I use a number that's too large so let's actually reduce this to to this and now you can see that the Fibonacci generation uh with the list took 0.28 seconds and 446 megab of ram the generator version took 0.1 seconds and 0.12 megabytes of ram because everything is evaluated in a lazy way everything is loaded on demand and also this is not limited in any way I can always get the next next next next next number um easily by just calling next or by just iterating further now another use case of uh generators is for certain data structures and the file that I have here this binary search tree is actually the same implementation here from um from the video so this is from my data structures tutorial Series in Python this is the exact same implementation and if you have a data structure that you're implementing and you want to implement this iter Dunder method where you iterate over it so if you want to be able to do some something like uh creating a binary search tree and then saying for element in BST if you want to be able to do that with your own custom data structure you need to be able to implement this uh iteration Dunder and this is of course possible with a yield in this case with yield from because I have these other uh private methods here that are yielding the individual key value pairs here but for some data structures you're going to need generators as well so I would say that these are the main reasons another reason could be in general working with stream data working with remote data getting things on demand getting things step by step but the main reason to use generators other than if it's needed in a data structure or if you needed to represent a an infinite sequence the main reason to use generators is the efficiency when it comes to your resources less CPU Cycles less Ram needed everything is just more efficient and doesn't have to be loaded all at once now you should not always use generators of course because sometimes you want to have all the values stored and you want to access them quickly by using an index but if you're going to process them step by step anyways you can also use a generator so that's the main reason to do that so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you in the next video and bye