Python's Magical Itertools Module

Key Takeaways

[Music] in this video i'm going to be introducing you to an awesome module in python called itter tools now this module is indeed magical and the reason for that is it provides a bunch of built-in functionality specifically functions for creating iterators for efficient looping if i'm just quickly scrolling through here you can see we have stuff like count cycle repeat accumulate chain chain from iterable compress pairwise star map product permutations combinations and a ton of stuff that you've probably done by yourself from scratch simply because you did not know the function existed in a built-in module called itter tools so with that said let's head over to the code editor and i'll start walking you through some of the more useful functions from the intertools module so the first thing i'm going to do here is give you a very quick explanation of what an iterator is because all of the intertools functions are going to return an iterator object and we should probably understand how that works now i will mention that if you do have my programming course programming expert then you would already understand iterators you'd understand generators decorators more advanced programming object-oriented programming and all kinds of cool features in python so if you're interested check it out from the link in the description programming expert dot io you can use discount code tim anyways what is an iterator well to demonstrate this to you let's start by having a look at something known as the range function now we've probably seen this before but if i do something like r is equal to range and then let's go with a range of say 1 to 10 when i print this out here let's have a look and notice i simply get range 110 now range itself is not an iterator but it's known as iterable now anything that's iterable simply has an iter method on so iter is this function that you can call and what this does is actually call the corresponding underscore underscore either which is on the class that represents this range function so i know this is a bit complicated but on range there is this underscore underscore either function that you can call when you call that it returns to you an iterator and the iterator is what you iterate over to get all of the values in the range created by this object so for example if i print out the iter of r you're going to see that i get a range iterator object now this range iterator object is something i can iterate over so let's have a look here let's just say i is equal to that and now let's print out for example the list of i when i do this i get one two three four five six seven eight nine because this is iterable so i can use the list function on it and it gives me all of the elements in this iterator however an iterator is an iterator when it has an underscore underscore next underscore underscore method so this method is a special method that returns to the next value in the iterable sequence so when i have a range of 1 to 10 what i can do is the following i can print out the next of my iterator which is i and let's print this out a few times now when i do this notice i get 1 2 and 3 right it's giving me all of the values in my sequence and then to go even further let's have a look at something like this 4x in and then i'm going to say i because i can loop through an iterator right and i have my iterator i and when i start printing x here let's have a look and notice that i don't get any repeated values so i have my one two three from all of the necks up here and then when i continue looping through the iterator i get four five six seven eight nine because since we've called the next method manually we've been moved to the next values in the iterable sequence i know this is a little bit confusing if this is the first time that you're seeing it but essentially when you have an iterator or something that's iterable sorry you can call this iter function on it that returns to you an iterator and then the iterator you can call this next function on and it gives you the next value in the iterator sequence you know that you're finished iterating when a specific exception is raised this exception is known as stop iteration and to show you that let's just print next a bunch of times so we only have nine values in our sequence here so when i print the next of i more than nine times you'll see here that we get a stop iteration exception and that tells us there's no more values in our iterator to be returned so that was a very high level overview of iterators and iterable objects however you may be interested to know that when we have a for loop like this and we're looping through an iterable object what actually happens implicitly is you're going to call the iter function or method on this iterable object that's going to return an iterator and then the for loop is going to continually call the next method on this iterator until eventually the stop iteration exception is raised so once that exception is raised the for loop knows there's no more objects to loop through and it can stop iterating and the value that you're going to look at for i here for our iterator variable is going to be equal to whatever that next method returned at every single step in the iterator hopefully that makes a little bit of sense but again we have the next function we have the inner function and these are really just mapping to the dot underscore underscore next like this and to the dot and then this is going to be underscore underscore iter underscore underscore method so anything that is iterable has this iter method and anything that is an iterator has this next method and when you use the next function and the iter function you're just calling these methods on those objects all right hopefully that is clear now let's have a look at the itertools module so the first type of functions i'm going to show you from the editor tools module are known as infinite iterators and what that means is that they can return an infinite sequence and they actually do this by using something known as the generator syntax i'm not going to explain what a generator is in this video again you could learn that from something like programming expert but for now let's have a look at the first function which is count so this is an infinite iterator and the way this works is you pass a start and an optional step value if you don't pass a step this will by default be one and this will do exactly what it says it will simply count a sequence starting at some value and going up by the step every single time so if i start at 10 which is what i'm passing for the start here and then i have a step of five we're going to count up by five until we decide we no longer want to iterate over this iterator so if i run the code here you can see that we get a sequence where we are starting at 10 going up by five and i've just made it so we manually stop here at a hundred if i didn't have this break then this would happen infinitely because again this is an infinite sequence so that is cool that is the count function now let's move on to the next one the next function that i have for you is also an infinite iterator and this is known as repeat now this is pretty straightforward but this is going to take in a element this element could be a list it could be a string it can be really anything you want and then the maximum number of times you'd like to repeat this this is optional if you do not pass this it will repeat indefinitely so for here if i have repeat and i'm repeating hello and i say a maximum of 10 times then i can loop through this repeater iterator that's what's going to be returned here when you call the repeat function up to 10 times and print out the value so if i do this you'll just see that we get 10 hellos showing up on the screen so this is useful if you want to repeat something a certain number of times so the next function that i have for you is known as cycle now this is pretty straightforward this is going to take in some iterable object could be a string could be a tuple could be a list really whatever you want and it's going to allow you to cycle over all of the elements as many times as you would like again these are infinite iterators so you have to decide when you want to stop iterating over them they will continue to generate a sequence forever so let's run the code here and have a look at what this does notice that we just cycle through a b c d e f constantly until i have a manual stop here where we've done this over a hundred times so what i'm actually doing is manually calling the next method here on my cycler which is equal to a cycle iterator and if we want to just have a quick look here at what the iterator object actually looks like we can print out our cycler and notice here that i get an itertools.cycle object which is an iterator and we know it's an iterator because it has this next method which i can manually call using the next function now i also could loop over this using a for loop if i wanted to so i could just do something like 4i in and then this would be cycler however if i did this i would need to implement a manual break condition because again that would go infinitely unless i manually broke it at some point so that is cycle again takes in some iterable and allows you to cycle over it as many times as you would like so the next set of iterators that i'm going to show you are known as terminating iterators they're not infinite they do have a defined number of elements that they will return to you so the first iterator to look at here is known as accumulate and the best example of this is something like a running sum so i can say the running sum is equal to accumulate and then accumulate an iterable object in this case a bunch of integers and what this does is give me the sum of all of the elements at the current position and prior in the list so if i print this out here you see we get 1 3 6 10 so on and so forth so the first element is the sum of just the first element which is one second is the sum of these two elements the third is the sum of these three elements so on until you get to the very end and one thing to note here about this is that you've probably implemented this behavior by yourself in python before this will be much more memory efficient as well as faster than if you were just just to write this story with your own for loop and again the reason for this is it's going to use the generator syntax it's actually only going to give you these values when you request them it's not going to be storing them in memory and taking up unnecessary space one other thing to note here is that whenever you're looking at an iterator so in this case we have accumulate we can call the list function on it directly and that will run the next method on the iterator until there's no more elements left and just collect all those values in a list so rather than manually looping through this using a for loop we're calling the next method ourself we're just using lists now to quickly grab the results of the iterate hopefully that makes sense that is accumulate so the next iterator to show you is known as chain this is very straightforward it simply chains two iterable objects together if i run this code you can see that we get abc and then def where the first iterable was this string and the second iterable was this one again this is going to be more performant and memory efficient than if you were to try to implement this behavior on your own and say concatenate two lists together this is not concatenating two lists it is returning to an iterator that allows you to retrieve one element at a time and process and use that as you need to that is the point of the iterators is that you're not storing everything in memory you're grabbing one individual element at a time and using it as you need to be as opposed to storing everything in memory when you only need the current element in the sequence hopefully that makes a bit of sense let's move on to the next example the next example is very similar to the first one but this is chain from iterable now let's just have a look at what this does when we pass in a nested list so notice here that i actually get this list flattened so this is something that you can do with the chain from interval pass in some type of nested structure it will actually flatten that structure for you by chaining all of the elements that are inside of this iterable object next we'll move on to compress as i was saying the next iterator that we have here is known as compress now what this does is take in some data source as well as some selectors and it simply keeps all of the items that are inside of this data if the corresponding item in the selector is true so in this case i have a nested structure that has three elements so this this and this then i pass in an array here i guess sorry a list that contains booleans now true indicates that we're going to keep an element false means we are not going to keep it so if i run this you'll see that we only keep a b and c because true was here for the first element now if i make this true you'll see that now we get the second element as well and i could alternatively change these to be anything that returns a truthy value so i could put say zero and one and this would work as well moving on we have the pairwise function this one is cool it's simply going to pair all of the adjacent elements in our iterable so let's run this and have a look and notice we get one two two three three four so on and so forth moving on the next set of iterators i'm going to show you are known as combinatronic iterators now the first one to have a look at here is a product now what the product is going to do is return the cartesian product of two iterable objects now this would be equivalent to kind of a nested for loop looping through every single possible pair of the items in iterable one and iterable two so let's have a look here when i print this out notice i get 1a 1b 1c 2a 2b 2c 3a 3b 3c that is what the cartesian product returns moving on we have the permutations iterator or function whatever you'd like to refer to it as and what this does is return all of the permutations of a particular size of an iterable object so let's just have a look at what this returns we get a b a c a d b a b c so on and so forth and notice the order here is important so something like bc and cb are different permutations because the c and the b are in a different order that is not the same as what i'm going to show you next which is combinations so as i promised the next function to look at here is combinations and this is going to give you all of the combinations of a particular size of an iterable object so let's have a look at this and notice these are the combinations note there is a lot less combinations than there is permutations and the reason for that is that the ordering of the elements does not matter in a combination so ac and ca would be the same combination hence why we're getting less pairs here now let's have a look at a size of three notice we're going to have four possible combinations here if we have a look at four we're only going to have one combination because there's only one unique combination of a b c and d so with that said i will start wrapping up the video here i will mention that these can become very powerful when you start combining them together so combining say the combinatronic iterator is with a terminating iterator maybe mixing in a map and filter function somewhere along the line and there's a bunch of other functions here that i did not show you from the intertools module so feel free to have a look at those from the documentation in the description hopefully you guys found some value from this video if you did make sure to leave a like subscribe to the channel and i will see you in another one [Music] you