Expert Python Tutorial #5 - Generators

Key Takeaways

hello everyone and welcome back to their expert Python tutorials so today's video we're gonna be talking about generators and generators are a pretty straightforward and actually very useful expert level feature in Python and chances are you've probably seen them before now before we get into generators I do need to quickly mention that this series has been sponsored by kite so kite is the AI autocomplete for Python that I've been using throughout this entire series and just through my daily coding you can see that when I actually start typing something we get these nice suggestions top popping up from kites with little kite icon here it is just the best Python autocomplete it works very quickly and actually saves you quite a bit of time and works for other modules that aren't just built in Python things so it's pretty useful if you guys want to download it there's a link in the description and it's free so you know go ahead and do yourself a favor if you do want that autocomplete ok so now let's get into generators now actually before I discussed Jenner's I want to show you the problem the generators help us solve so we should understand that in our computer we have a finite amount of memory a finite amount of RAM and when we run a computer program our program is loaded into that memory and when we're storing things manipulating variables lists all of that that's well stored in our computer's memory that's because that is the fastest way for us to retrieve and write to data while we're in a program you know we could put some things on a hard drive but that's pretty slow and it really depends on what we're doing but most of the time we're working in RAM and what that means is that we are limited to the amount of RAM that is in our physical computer when we're writing a computer program and in fact you know most of us will never even see the error that I'm about to show you but it is possible to actually fill up all of RAM right to use all of RAM to have nothing left and in fact that's what I'm gonna try to do right now so if we look at the example on the screen and I just started running the code you're gonna see this takes a second to run and what I'm trying to do is generate the sequence of all of the squares from the number 0 up until I want to say this is like 100 million or 10 million or some number like that right and then what I'm gonna do is loop over all those numbers and print them out and you're gonna notice that very shortly yes and there we go we get the error we get a memory error which essentially means that nope we can't do this these numbers are too big we don't have enough room left in memory we're not to use that much space no that's not allowed so how do we go ahead and how do we fix this well this is where we use a generator now before I actually create a generator I want to look at something here if all I'm doing in this specific application is just printing or processing say one value at a time which is what I'm doing when I'm looping over this list right one value at a time I'm processing and I'm not accessing other values I don't need values that are in the future I don't need values that are in the past then is it really necessary for me to create a list like this to loop through and the answer is no obviously right in fact what I could do is replace this for loop with just a print statement right so I could replace this entire thing actually we'll just do it down here let's say for I in range and then we'll just type that same number whatever that's close enough and then what I can do is print by the exponent two like that right and this is the same thing and this is the idea behind generators this is not what we're gonna do precisely to generate this sequence but generators allow us essentially to look at one value at a time and to not store the entire sequence of numbers when we don't need to do that and I'm actually going to code out our own kind of generator pattern here that just uses a class and some dundar methods to illustrate what's actually happening inside of a generator when we use it now I understand that most of you probably still don't know what a generator is but after I write this class and start showing you hopefully this should make sense so what I'm gonna call this is just Jen and what I want this to do is the exact same thing that we did before we're just gonna generate a sequence of all of the squares up to some number let's say that numbers then so in this case I'm gonna say let's define underscore underscore init underscore underscore at let's take self let's take and let's say self dot n is equal to N and then let's say self dot last is equal to zero so this is gonna be the last number that we generated the square for and we're gonna use this variable to keep track of the last number we generated the square for so that we know which next number to use and I'll show you how this works we're gonna define a actually in this case we're going to do dunder method so underscore underscore next it's gonna take self and what its gonna do is return self dot next now I know we haven't defined self-thought next yet what we're gonna do that now so define next self and didn't hear what I'm gonna do is generate that sequence that I did before except I'm just gonna do it using these two variables I'm not gonna do it using a for loop like we had before and storing everything in a list so what I'm gonna say is our V equals self don't last to the exponent two so this is saying our return value is gonna be equal to whatever this last number is that we have to the power to right so we're gonna find the square for that then I'm gonna say self thought last plus equals one we're gonna say if self thought last and in this case equals equals self dot n plus one actually what I need to do is sir I put this at the beginning of my loop so if self thought last here equals equals self thought and then what we're gonna do is say raise stop iteration stop iteration is just an error that we can raise that essentially tells us hey no we can't go any further and then what we're gonna do is return this RV okay so now I'm going to show you how we can use this so this we can actually use in a very similar way that we were kind of looping through the numbers before and we're going to say in this case G equals Jen let's say we want to loop up to you know some massive number like that and now if I want to actually loop through this entire sequence what I can do and I mean something you're saying oh this isn't big enough okay let's go bigger we'll say Wow in this case I'm true we're gonna say try print the next value of G right and what this underscore R square next is allowing me to do is call this next function or method or whatever you want to call it on G here and we can print the value and then I'm gonna say accept stop iteration and then we're simply going to break this loop like that and we should actually be good with that okay so what this is gonna do now is loop and I'm just gonna make this number smaller just so we can see that it does actually just go up to a hundred um but I promise you this does work with an infinite sequence and we'll watch and we can see that this works just like we would have before and we can generate all of the squares for these numbers now I can make this number absolutely massive I can make it you know that's super long number and this would be totally fine and in fact that's because we're not storing all this in a list we're restoring all of the previous values all we're keeping track of is kind of the internal state of the next number we need to generate and this is the idea behind a generator is that we don't need to store every single value what we can do is just store you know almost the last value that we generated and then using that we can generate the next one right or we can figure out the next number to generate and obviously this kind of pattern like you can write this if you want but what Python has done for us is kind of come up with a pattern that makes us a little bit easier called the generator pattern so what I'm gonna do now is pretty much take this class I'm gonna convert it to a generator and I'm going to show you how this works so let me say define gen and here this will just take the value and what I'm gonna say in here say for I in range and in this case we'll say and then we're gonna say yield I to the exponent - now I know this seems weird but essentially the way that this works is we instead of using this crazy pattern that we had before we're gonna use the yield keyword now what the yield keyword does is as soon as we hit this it returns this value until wherever this was called from or wherever we were looping through and then it pauses this function so rather than stopping the execution of this function which is what a regular function would do when we hit the return keyword it just pauses it which means we actually keep track of what I was so we still know what that number is we still know you know what n is we still have all of the internal information of this function to store it in memory we haven't gotten rid of that but we're just pausing when we hit this yield keyword that's what that means so think of yield as like a pause whereas for example return would be a stop of execution so we can use this generator by doing something like this G equals Jen and give it some number like this and then what we can do is say for I in G and then what we can do is simply print out I now this is gonna work the exact same as all the other examples I've seen I've shown you before and you can see that indeed it is working and we could run this you know infinitely and we're not actually gonna ever run into a problem and in fact I probably should have made this number a little bit smaller because I feel like this is gonna run for a long time but anyways that can go down there so this is the point of a generator and I know some of you are still confused but the way you have to think about it is we create this generator by creating this function Jen with what do you call you know hundred thousand or whatever it is whatever this number is 1 million and then what we can actually do is loop through this and what happens when we loop through this is it runs this for loop up until it sees this yield keyword so it sees this yield keyword i 2 squared right and then as soon as it sees that what it does is it pauses the funkiness is okay we don't need to run this anymore and then it waits until it's called again so until we loop through it again with this for loop the next method on it is called and then it returns to us the next yield and because it's pausing rather than stopping execution this is totally fine and in fact we can actually look at how much memory this function is using versus if we were to do the same thing say with a list and we'll do that in a second but I want to show you that to use this generator I don't have that just loop through it with a for loop I can use that next method that I showed you before and if I do next G and I run this you can see we get the 0 because obviously the first answer is 0 and if we do this a few times we get 0 1 4 9 so this is how this works when we call the next method or the next function on our generator we get the next value so essentially up until it hits the next yield keyword and in fact we actually can use more than one yield keyword in our jann if we want and I could say yield 1 and we can do as many yields as we want so let's do this and then let's actually show what happens when we go past how many yield keywords we have so if we look at this here we can see we get 10 a hundred thousand ten thousand and then we actually end up getting air and there was a one up here just you can see and that error is actually a stop iteration air right here because we don't have any more keywords yield inside this generator so there's nothing more to return so this works the exact same way that I showed you with that class that I did before where it keeps track of the internal state and knows what line it was on when it paused and then it goes to the next one until it hits the next yield keyword and that's when it pauses again and that's how a generator and they're pretty useful there's some other things that we can do with them and in fact I'll show you comparing the size of say a generator like the one we just created versus actually generating an entire list so you can see in memory how much this is actually using so if we want to generate the sequence right we know this generator works we can use that now so let's run that and what I'm gonna do now is import sys I'm gonna actually make that list that I did before so psi X equals and in this case we'll say I to the exponent two for I in range and we'll just do let's say 10,000 like that and we'll replace 10,000 here and now what I'm going to do is just print out the size of both of these so I'm gonna say sys dog get size of and in this case the first size we'll look at is X which is that list and then we'll get the size of G and I'll show you the difference in memory so what this does is tell us how many bits are actually I believe how many bytes are being used by whatever object is we pass in so in this case the list and in this case the generator you're going to see that we have a substantial difference so this first list is using forty three thousand eight hundred sixteen bytes to generate and store the sequence whereas our generator is only using 64 so you can see that this is really a crazy optimized way to be able to generate say infinitely lank infinite length sequences and when we don't need every single value in the sequence at once we just need it one at a time which is what we're doing when we say loop through something then it's much better to use a generator to generate this sequence for us now there's a lot more that I can show you with a generator we can close a generator we can stop a generator we can send values to a generator but I'm gonna leave that for people that are a little bit more advanced the point of this is to introduce you to this concept of the generator whereas when you're programming think about say if you're gonna make a massive list like this with a bunch of values do I need all of these values am I using one value at a time am i looping through my printing it am i adding it to something or do I only need say the last value and this value or a few values and obviously you can make your generators more complicated than a simple for loop you can do some more complex can't be you can start variables inside of here because remember that this yield keyword just pauses the execution when this generator is called and what that means is that I can loop through this this first generator say you know five times and then somewhere later in my program continue looping through it and it will you know resume where it left off and that's a useful kind of construct it's hard to show any examples in this you know short amount of time but hopefully this gave you an idea of when you would use a generator so with that being said I hope you guys enjoyed if you did make sure you leave a like subscribe to the channel and as always let me know if there's any other expert-level features you would like to see in the comments down below