Exploring Asynchronous Iterators and Iterables: a Basic Example, aiter and anext

Key Takeaways

Hey and welcome to this course where you and I will be exploring asynchronous iterators and iterables. Once you are familiar with the basics of the Python programming language and you have proudly started writing code that actually works, you might start thinking about making that code work more effectively. And with that I mean use the CPU more effectively. So you might have read up a bit on async programming and given that you are interested in this course I'm kind of going to assume that you have and you might realize that async programming without loops or without iteration doesn't get you very far. So that is what you and I will be fixing in this course. Ideally you will have heard of async programming before. So this course could be considered your next step on your async programming journey. Having said that though, there is a quick recap of the async basics at the start of the course. Now, you will also benefit from some basic knowledge when it comes to classes, attributes, and methods. And lastly, it's also best if you don't mind a little bit of silliness when it comes to the example that you'll be working on. Why not have a little bit of fun during our time together? So in this course you'll learn what async iterators and iterables are. How concurrency is achieved using async iterators and iterables and then how async iterators and iterables are implemented. Now more specifically you will start with a basic async programming example. So one that does not include iteration. Then secondly there's the introduction to our async iterators and iterables. And then thirdly, there are examples. So that's what's on offer. My name is Steven Loyens and I will be your instructor for this course. See you in the first lesson. Let's dive in, shall we? So in your first lesson, I'd like to talk you through a basic example of asynchronous programming. Now I'm doing this for two reasons. Firstly, this example forms the basis for the rest of the course. So I think it makes sense to start there. But secondly, it has the added benefit of being a quick recap of async programming before you learn about iterators and iterables. However, if async programming is completely new to you, then I include a link to a great real Python tutorial that will bring you up to speed in no time. And this is our example. Say you find yourself at home during the weekend and you need to check your email. So, you'll need to start up your laptop. However, you're also in desperate need of a coffee because it was a bit of a rough night last night and it just so happens that a parcel has just arrived and it's waiting in the hall for you to open it. Now, there are two ways in which you can tackle these three tasks. Firstly, well, you could just start up your laptop and sit there for 10 minutes. And once the laptop has started up, you can then go into the kitchen and make coffee, which will take 3 minutes. And then once the coffee is made, you walk into the hall and you open up your parcel which will take you another 3 minutes. So in total these three tasks will have taken you 16 minutes. And that is actually what happens when you code sequentially or synchronously. One task is performed after the other. It means that a task cannot start until the current task is done. So the current task is blocking the code. Now, a second way of addressing your problem is that you start up your laptop and while your laptop is starting up, you know, this will take a little while. You walk into the kitchen and you start making your coffee. Now, while the coffee is boiling and brewing, you walk into the hall and you open up your parcel and that takes you about 3 minutes, by which point your coffee is ready and all you have to do is wait for your laptop to start up. So in total this would have taken you 10 minutes as opposed to 16. Now to me this feels like a more natural way of addressing the problem and that is what you will try to achieve with asynchronous programming. So in this analogy I'd like you to think of yourself as the CPU. So while you were sitting waiting for your laptop to start up that was effectively the CPU sitting there being idle. So asynchronous programming multitasking involves a more effective use of the CPU's time. Now as a side note, you could have addressed this problem in a third way. Actually, while you were sitting there waiting for the laptop to start up, you could have asked a second person in the house to make the coffee and a third person to open the parcel for you. Now the result would have been the same. After 10 minutes, all three tasks would have been done. But this is not async programming. This is not you multitasking. You only did one thing. You sat and waited for your laptop to start up. You asking help of others. That is actually multithreading. Now I just wanted to highlight that because there tends to be some confusion between the two. So async programming only uses one CPU. Multi-threading uses multiple CPUs. In the next lesson then I will walk you through the code of this async programming example. So in this lesson I will talk you through the code of the basic async programming example and you will be using code timing. So please use pip to pip install code timing and also please remember that of course you can download the code so you can follow along more easily. And here is the code. So line one import async io that is the package that is used for asynchronous programming and then from code timing on line two import timer. Now remember I want to run three tasks concurrently. I want to multitask between three tasks. So I've created a generic run task function on line six. Now I want to multitask asynchronously. So I use async defaf and the input parameters are name and duration. The key functionality here is on line 11 where I use sleep from async io to simulate a task that takes a certain duration. So a certain number of seconds and I want to multitask. So I use a wait. Line eight. I create a timer which I then use on line nine to start a timer and on line 12 to stop it. So I can time how long the task actually took. So the tasks are in line 16. So in our task list, which has three tpples, one tpple for each task with the name and the duration. But then scroll down to line 19 where is our main function. Again, I want to multitask here. So it's async defaf. And the key functionality is really on lines 22 and 23. On line 22, I use gather from async IO to multitask between three tasks. And those three tasks are on line 23. So I use run task or I call run task three times. And each time I unpack a different tpple from the task list. Then on line 20, I create a timer which again I use on line 21 and 25 to time how long the whole thing took. And finally on line 30 I start the event loop using run from async io. So to run the file I type python async. py and when you hit enter you'll see that the three tasks have started at the same time. So start laptop make coffee and open parcel. and make coffee took 3 seconds as did open parcel that took 3 seconds and by now start laptop which took 10 seconds has also finished. So in total it took 10 seconds not 16 and of course that is what we were aiming for because we were coding asynchronously. So multitasking did happen. That's a good start but we haven't done any iteration yet. So in the next lesson you will be looking at iterators and iterables. So now that you are familiar with async programming you're moving on to the introduction to asynchronous iterators and iterables. Now to start with though forget asynchronous programming for a second and just focus on the concepts of iterators and iterables. Now what is an iterable? An iterable you can think of as a container of data that can be iterated over such as a list for example. And what is an iterator? Well, an iterator you can think of as the object that controls the iteration process over your iterable. Okay, so what does that mean in terms of implementation? Well, in terms of implementation, an iterable implements the it special method. An iterator then implements the it and the next special methods. Now, if you need a little bit more help with this, I include a link to a real Python tutorial that explains iteration in a bit more detail. Now, if you bring back the idea of async programming, you then get the async iterators and iterables. So, an async iterable holds data to iterate over asynchronously. An async iterator then controls the iteration process over such as synchronous iterable. And in terms of implementation, the async iterable implements the aitter special method. And an async iterator implements the aitter and the a next special methods. Okay. Now to bring this concept a bit more to life, I'd like you to think back of a list. a list which you know is imperable. So when I invite you to open up your ripple and apply dear to the list object and when you hit enter you'll see a whole array of methods and special methods one of which is the it special method. So the list object is an iterable because it implements the it special method that is what that means. Now if you look again can you find the aitter special method? Well I can't say it. So a list is not an async iterable. It's an iterable but not an async iterable. You're going to have to build your own. So in the next lesson you'll cover the implementation of the aitter and a next special methods. In the previous lesson you learned that an async iterable implements the aitter special method and an async iterator additionally implements the a next special method. So it makes sense to start with the aitter special method and have a look how to implement that. So an a iter special method must return an async iterator. But hang on the aitter special method is implemented by an iterable too. So are those two related? Well, yes they are. You can think of it this way. An iterable holds the data that can be iterated over. And to then actually iterate over your iterable, you call the aitter special method on the iterable to give you the object that controls that iteration process. And that object as you know is your iterator. So let's then move on to the iterator which additionally implements the a next special method. So the a next special method must return the next element of the async iterable. That makes sense. It also must return an awaitable object. I guess that makes sense cuz we're working asynchronously. And it must raise a stop async iteration exception when the data stream is exhausted. So you can see how the implementation of a next controls the iteration process. Well, firstly it provides the next elements in the loop and then it stops the loop when there are no more elements to iterate over. Okay. So how do you use these things then? Well, you use them in asynchronous loops as you might have expected. So async for loops for example. And then in terms of use, I'd like to finally remind you of the key point of async iteration. And that is to provide nonblocking iteration. So what that means is that the running of an async loop such as async 4 doesn't block the code from running other async tasks or loops. So to hammer these concepts home and to see how they are being used in practice, you are going to introduce async iteration in your code example in the next lesson.