Exploring the New Features of Python 3.12 | Real Python Podcast #175

Key Takeaways

welcome to the real python podcast this is episode 175 python 312 is here a regular guests G hella and Christopher trau returned to discuss the new version Gana coordinated a series of preview articles with several members of the real python team this year and his annual piece published on Monday October 2nd titled python 312 cool new features for you to try and Christopher's video course posts the next day covering the topics from the article with visual EX examples of python 312 in action G and Christopher collaborated to create code examples of the new features we discuss better error messaging more intuitive FST strings sub interpreters the Linux perf profiler improved typing syntax and more we dig into the updates and offer advice on incorporating them into your projects we also consider when you should start running python 312 all right let's get started [Music] the real python podcast is a weekly conversation about using python in the real world my name is Christopher Bailey your host each week we feature interviews with experts in the community and discussions about the topics articles and courses found at real python. after the podcast join us and learn real world python skills with the community of experts at real python. all right well it's python 312 week here at real python and as has been an annual tradition we have GNA hella and Christopher Trudeau back on the show so welcome back GNA thank you Chris H this I like this transition it's fun to hang out yeah it's very cool and Christopher Trudeau is back again I try not to stress the again in that I'll hit it really hard so we've kind of divided up some of the features that are part of python 312 and going to go through them kind of back and forth and discuss them Gana has worked on a preview article that came out earlier this week kind of been an annual tradition also previewing a lot of the features that are in it along with that Christopher just finished his video course that covers a lot of the same features and so those are both available on the site and I'll talk about them a little bit more in the show as the video Spotlight this week one thing that is a little bit different is this year Gana continued the tradition of having preview articles getting people prepped to understand what's coming out with 312 this year it sounds like you you had several more preview articles and maybe some help is that right uh right yeah so we've been I guess we started this last year so for the 311 we started doing a few in-depth articles about certain features and last year I wrote I think we did three preview articles plus the full article but this year we got the whole team involved so this year we actually did five of these preview tutorials about the different features so we'll add some links to all of those as well where we kind of go really deep about the new features yeah and then in this this week's uh tutorial we kind of more just show the highlights of each other features yeah and we'll get a chance to drill in a little bit into them and then also add our commentary on the different features as we go but yeah the those are nice deep Dives and um we'll also get a chance to highlight some of the additional authors there y so generally that worked well do you feel like you're gonna do that for the next release not to preempt any of that for you right I guess we haven't talked about it within the team but my feeling is that was uh yeah very successful and I think it was fun to involve the whole team in this and we kind of got to play off each other a little bit more and and so on and it's the the preview articles are are they're good for me as sort of like Research into the full article okay and I think it's good for all the the readers as well for kind of having the option to go in depth for the features they are interested in nice yeah and I think they I think they were particularly important this year because so much of the 312 stuff is sort of internal right when we did the course we're talk briefly about things like sub interpreters most people aren't going to interact with these things you get a couple slides that say this is what it is but if it's if it's the thing you're interested in then you've got this depth you can go off and uh you can dig in as much as you want so it I think it it works really really well for uh different levels of interest in the audience yeah you can kind of pick and choose a little bit and and drill in on stuff you get a real chance to to focus on things that may be of more interest for a particular Audi audience so you guys ready to dig in then to some of the features we're going to kind of go back and forth between G and and Christopher on on covering these things sure all right so garan uh tell me how we've been continuing to uh get better and better error messages uh right yeah so as uh you may remember uh in Python 310 that there was kind of the start of initiative of of rewriting many of the error messages especially syntax errors yeah this was something that was kind of made possible by the switch to the new parser where we could kind of be more precise with things so in 310 there was a lot of error messages were improved they got more informative and they got also more precise so they could actually point out the line where something bad was happening for 311 it was slightly different where they rather improved the trace packs themselves and added these fun little squiggles into the trace packs in 312 it's back to just improving some of the error messages and in particular several of them relate to Imports and things like this so they can kind of nudge you towards did you forget to import some standard Library module or did you mess up the syntax were importing and and those kind of things so it's just many of these small the user conveniences or life improvements that you're kind of getting from this where the messages just become much easier to to work with so that there's I think three in particular about Imports so so one is just if you if you try to use something in a standard Library module and you don't import it it will essentially just say did you forget to import whatever module you used so say if you just type math.pi it will say did you forget to import math and then there's this thing where you can import just certain names from from a module or you do something like from math import Pi but if you mess up the order of this and say import PI from math which kind of sounds natural then it will there and use from math port instead I think Christopher was saying before that that is the way sometimes it's worded in other languages oh yeah that might be yeah yeah it's a common mistake for people coming from other places yeah and the Third Kind of related improved error message they have is that you if if you're done doing one of these import certain names and you misspell the name it's you usually able to tell what What You misspelled so for instance if I do something like from math import Pi but I spell it py it will ask me did you mean pi as in pi instead or things like this so okay it has to be within the ballpark of um what it could guess I guess yeah and this is something they introduced in 310 where they had these suggestion features so it's doing uh some kind of fussy string search essentially on the other the available names so yeah if it finds something that's fairly close then it will use it as a suggestion uh the the final error message they have improved uh relates to classes and if you're using some instance attributes so typically you have self. something uh but you forget the self do part uh it it's usually able to just ask did you really mean just to say say radius or something or did you mean self. radius again it's just nudging you in the in the right direction to kind of help you see at a glance what's wrong yeah it's these are such quality of life improvements for beginners and intermediates or anybody who makes typos right for everybody quite quite frankly it is the most used feature for me of 310 and 311 and I suspect it will be for 312 I I I know the walrus exists I know they added it I've never used it but did you mean shows up pretty much every day of the week I code yeah yeah so yeah so so they just just continue the nice work of of improving this I suspect we'll see more of these also in 313 and later um and uh this was one of those things where we did an in-depth feature on yeah so the python 32 preview we call it ever better error messages since this seems to be the continuing Trend and in that one Martin Bryce was the author and he did a really nice Deep dive into the C code to kind of show off a little bit about how this suggestion feature is implemented and where it kind of finds suggestions and so on so if if you're interested in just really digging into what it takes to create these error messages uh that preview article is really nice yeah I talked for a while with Pablo Galindo Sado about this back in I think with 310 was coming out and it was really fun because we kind of were able to dig into a little bit how the peg parts are work and I believe he's still pretty deeply involved in this like this is one of his uh his babies in this whole project so yeah Pao I think has implemented all of these improved eror messages also now for 312 so awesome so Chris tell me what's going on with FST strings this year so F strings have been around for a while now they were actually introduced in Python 36 and that's before so gar already sort of made the mention of the new parser the peg that showed up in 39ish 310 depending on how you want to draw that line and so when they introduced FST strings into python this was before that parser and the parser they were using at the time couldn't actually it wasn't powerful enough to do F strings so what they did was they had the parser identify it as a string and then pass it off to a secondary parser and that secondary parser would actually do the f- string parsing well the new Peg or it's not I guess it's not that new anymore but newish Peg relative is capable of parsing the F strings and more so what they've done in this release is essentially to replace the parsing code with Peg parsing code this primarily means less code for the maintainers and a cleaner experience on the back end but it has some side effects that are beneficial so when doing this change a few of the limitations that the old specialty parser had were to be removed so for example the first one is how nested quotes work so with the old parser an FST string had to follow the same rules as a regular string which means if you wanted to use quotes inside a brace expression they to be different kinds of quotes than the surrounding ones right so double quotes on the outside and single on the inside say so particularly like if you were trying to get at like a value inside of a dictionary yeah and so this restriction has been removed so in 312 what's inside of those brace brackets is treated as a separate idea than the string itself and so you can use double quotes inside of those brace Expressions if you like I'm not really sure I like this uh okay I part of that may just be I'm old and change bothers me I don't know but I'm so used to what a string looks like that this just doesn't scan for me but you know maybe I'll get there eventually one side effect of this is it's breaking linters and syntax highlighters as there's really no way to differentiate between 312 code and earlier code just by loading it they're either going to have to put like a flag into the linter or the linters aren't going to be able to identify the old problem anymore and we've seen it even the uh the automatic Checkers we have for the code in articles for Onre python had to be uh fudged a little bit to get the Articles to publish because of exactly this problem so another restriction that's been removed uh kind of similar is you didn't used to be able to use backslashes inside of those brace Expressions I didn't even know this restriction was there I'd never tried it before but the example given in most code is say you wanted to do a sln join uh inside of a brace expression uh before 312 you had to do it outside put it in a variable and then use the variable inside uh now inside you can do it okay another one what I'm sort of like okay whatever you can also now Nest F strings I'm not sure what the use case for this is uh but if you want F strings in your F strings hey go for it the commonality across all of these really is the idea that what's in the brace expression is now equivalent to an actual python expression rather than being this weird little subset that it was before four and so you essentially you get most of the power of the language inside the brace and don't have to think about it as being a special case yeah the one last restriction that got removed and this one I can definitely get behind is you can now put comments inside of the brace Expressions so this is particularly handy if you're doing say like a triple quoted multi-line FST string and you want to put a comment in the brace explaining what you're doing for example why you wanted to Nest a bunch of f strings together you should put a comment on that right well now you can do that and that's that's a nice little benefit yeah the fact that it can kind of do some code inside of an f-ring it might be nice to have that commented like why you're doing it there and I could think of some other like compilation thing where like some function is tying a bunch of strings together and that is maybe why they got nested I don't know but uh that that is interesting yeah I could see there's probably extreme cases of things like building reg x's and stuff like that where you you might want to Nest something I'm sure there's a case for it uh I'm not I'm not it sounds like I'm down on it I just I don't expect I'll be using it anytime soon but uh it doesn't hurt anything for it to be there and like I said I think it really is just sort of a a design decision side effect of going hey this is now an expression and therefore it can parse like any other expression rather than I I just I'm guessing but I doubt the core maintainer sat around and said you know what we really need a nested F strings so I suspect it's just a a benefit of how things have changed cool so G there's a article on this one too right a deeper dive yes exactly so lean is POS has written the preview articles on more intuitive and consistent F strings and I think I I've read somewhere that Pablo was involved in this one as well Pablo Galindo Sado the Cur um on the FST strings and I read somewhere that this nesting thing it's definitely not something they tell people you should really be doing this okay it was or they just figured out we don't need to restrict it and then Pro probably some lters will jump in and tell people please don't do this all right kind of just moving the Restriction out here's the really powerful gun please don't don't point it at your toes there we go nice so Gara you wanted to talk about a couple optimizations one being the comprehension in lining all right so I guess last year for 311 they all the bus was about the the faster cpython project and how P python was getting faster yeah and that's been in a sense more quiet this year for 312 and I think the 312 is definitely faster than 311 but not with the same amount as 311 was faster than 310 uh but the project is is continuing with several people working full-time on it there's a couple of things I'll just highlight one one of them is kind of the Highlight from last year was the specialized adaptive inter interpreter okay which essentially just means that they improve The cpython Interpreter so that it could change and adapt the bite code which is the code that's actually running during execution so that it would kind of notice things like okay you you keep multiplying float numbers so instead of running a regular multiplication operation we'll do the float multiplication operation which is faster okay so it kind of specialize things like this well it it was introduced last year and then it's just been improved for 312 and of course when you just improve stuff you don't get the same headlines uh they've done some good stuff there the process of using the specialized adaptive interpreter in 311 was that it had a quickening step and then a specialization step and the quickening step was essentially The Interpreter just looking at the code being executed and at some point noticing okay this this bite code keeps being executed so we we'll mark it as a candidate for specializations we kind of just notice The Interpreter in the future you should do something with this for 312 they have just removed the quickening step and just all the bite codes are available for specialization so it kind of simplified that process a little bit and then uh for the specialization part they just added a bunch more bite codes that can be specialized so they kind of started with 311 with sort of like the simple things like if you're just adding some numbers then you can add two Floats or you can add two integers and the operation internally is quite different so so they could kind of specialize for each of those and they've just been yeah adding many more of these specialized operations for 312 then the other one I want to talk about and this is kind of kind of the the optimization thing that got its own pep this year uh pep 709 um it's about inlining comprehensions so it's about list comprehensions dictionary comprehensions set comprehensions uh I don't think they include the generator generator Expressions this time okay uh but the other comprehensions or the regular comprehensions and this is again mainly just an internal thing but um when python sees the list comprehension what it actually does is that it rewrites the comprehension as a nested function and then into the bite code it kind of insert this function call to to call the comprehension function that it just created that's been the case now up until python 31 for python 312 it's inlining the comprehension instead which means that it doesn't make the nested function it just pops all the code into wherever that comprehension is called the nice thing with creating the function is that it makes things very clean and and and simple in terms of keeping the name spaces separate and and so on so that your variables inside of the comprehension don't leak okay out so you can have an X you can have a variable say name that's outside of comprehension and then you run your comprehension and you have a name for nameing names inside the comprehension and that name doesn't overwrite yeah the one that's on the outside okay that comes for free when when the comprehension is a function because that's how functions work you have their own they have their own name space uh so so when they did the inlining they kind of need to do some some tricks to make sure that they we still have the same behavior with the name spaces and so on but this they've been able to do so so they've then gotten these comprehensions uh to to run inline and the big uh bonus for us as users is that comprehensions are faster we see that the big difference for for short intervals because then you have most of the overhead for calling calling the nested function in the python 311 and I found some examples where I do really long intervals where the 312 was actually slightly slower uh I don't think that's because of the comprehension per se but that's some other operations have gotten slightly slower well others are faster and so on but what they've seen on the big benchmarks is that um they see a speed up of 10 to 15% on the on the general comprehensions okay so there should be at least some some bonus for this that there are actually few tiny changes uh that you might notice but this is essentially just how the local Nam space things changes and and if you get errors inside your your comprehension the trace back will be slightly different because in 311 you actually can see this nested function in the traceback and that will be gone now but in apart from that you shouldn't really notice any difference in 312 wi comprehensions so this is another one of those that's kind of in internal yeah under the hood under the hood kind of Chang this but it's it's part of these steps towards ever faster python yeah kind of digging further into more under the hood stuff Chris you want to talk about the uh support for sub interpreters yeah I think this is Hood's plural uh okay so depending on our listener's depth of knowledge excuse I'm going to go off on a little tangent here so in order to give the right background so Python's an interpreted language and let's contrast that to a compiled language so in a compiled language like C you're using a compiler to translate your code into the machine language that's directly used by your computer in an interpreted language like python it's got some sort of virtual machine mechanism that's running your script now it might compile it down into an intermediary bik code or something but it's it's running that and not the the computer's main language the advantage to this is scripts can run anywhere uh because as long as you've got an interpreter you can run it without it needing to be recompiled it's the interpreter's responsibility to talk the computer's language the disadvantage of this is tends to be slower because you're putting an abstraction layer between your code and the actual machine so when you type python on the command line you're invoking that interpreter and there's actually several different python interpreters out there and so when we say python we're actually being vague sometimes we mean the language and sometimes we mean The Interpreter right and The Interpreter that most people are using is cpython and so when G makes the comment that this is 10 to 15% faster what he's actually meaning is there were performance improvements in The cpython Interpreter that mechanism doesn't affect other third party interpreters right so so we're almost there most of the background so when you type python that first interpreter gets spawned and we're going to call that the main interpreter and that interpreter can create copies of itself and these copies are called sub interpreters and you do this when you create multiple processes for writing concurrent code now one problem that's common in current code is how to handle the situation where you've got two processes that want to modify the same value at the same time and if it happens in the wrong order you'll get bugs you'll get wrong values you might get a mix of the two values not just the race condition of who wins but like a muddled mess in between so to get around this in parallel coding we put locks in place uh to make sure that that kind of thing doesn't happen and this problem is so common that python has the lock that wins out over all other locks called the global interpreter lock which we've talked about probably to death on this show sure so this lock is trying to make sure that certain Global shared values aren't exposed to this kind of race condition now if you put a lock like this in place you're losing some of that parallelism because now everybody's waiting on that lock and so now to the core of what I'm actually talking about pep 684 and pep 550 4 uh both of these are about doing work with the sub interpreters and the Gill so pep 684 is moving the Gill into the sub interpreter and that means each interpreter will be able to work completely independently of all others so Gill actually isn't a gill anymore it's a sill uh because the lock is just at that interpreter level so this could mean a big boost to concurrency now the Gill's there to protect different extensions from stomping all over each other uh so a big part of this work really is changing things so that an extension can say hey I know how to use this and if the extension doesn't say hey I know how to use this then the old way is going to be used so that nothing gets broken we don't want to go through python 27 to3 again so pep 684 is all internal to see python pep 554 intends to actually expose all this work in the Python language and I say intens because this one is targeted for python 313 so they're making this change as a twep process so this is all somewhat related there's a whole buch of work going on right now in trying to get rid of the Gill or contain the Gill or use the Gill less and it's making python more capable with concurrent code and this is a big long-term ongoing project so if you write python code and only write python code you will not see a difference in this in 312 in 313 you'll get access some to some stuff that you wouldn't have had access access to before if you're using things like numpy when they get around to start taking advantage of this you might actually see some speed up and of course if you're an extension writer then you've got work to do yeah yeah I'm thinking that this groundwork that's being laid really is reinforcing the idea that if you are a package maintainer or you do those kinds of projects that are you know doing things that travel into sea or or into rust or something like that that you're going to really want to be testing these your code over the next year yeah for sure and the flip side of it is you know this is a sign of the maturity of the language right like it's starting to be used in such big ways that these things are becoming important right like python 1 was a little fiddly scripting thing on your local box and no one would have cared about this right uh now it's being used in huge multi-c concurrent beasts of software yeah uh and so we're running up into the limitations of the language and the core maintainers are doing a good job trying to figure out uh how how to get us to that next level and uh and keep the language growing and its usage to be deployed on on massive massive parallelism so we'll get there cool this one there was another preview article and this one was by a previous guest of ours uh Jim Anderson who'd been on the show hoping to get him back on the show actually talk a little bit more about maybe some of this stuff but that sounds like that one it came out like a a week or two ago right G yeah it's quite recent and that was first of all always great just have Jim back to to write an article for us yeah and he he did a great job of kind of writing about future python code so some of this Chris mentioned this for 312 but Jim also notes some some of the examples that should be possible in 313 or a later version of of python so it's a really interesting read yeah definitely and again if you're want to learn a little bit more about what's happening internally it's going to be a really good resource for you g do you want to talk about again this one has another company article with it talking about measuring performance especially if you're have Linux uh right uh so this is one of those things that you may not care about or it might be just what you needed so it's um well the short timeline is just that python now supports the Linux perf profiler to yeah give give a slight overview there so first of all a profiler is just a tool that you use to monitor the performance and you can diagnose the performance of your scripts and uh if you then profile your code uh then you'll kind of get some kind of report that will say something about either how fast things are running how much CPU things are using how much memory is being used and so on there are different kind of uh profilers yeah and python has long supported different profiling tools including a very simple tool called time it in the standard library and also C profile which is a regular profiler in the standard library and there have been some third party tools and so on so now the perf profiler that's one of sort of like the big players in Linux and it's actually built into Linux kernel so so it's it's really tightly integrated into the operating system if you've been trying to profile python code with perf what actually has happened so far is that it hasn't really been useful because perf will just tell you that well there's this C function called Pi eval eval frame default uh that running for 99% of your time so you it's all in this one box may may want to do something with that function exactly and that function is actually just your cpython interpreter that's just evaluating python code the the actual python code hasn't been available for perf it's just seeing the C program that you're running so it kind of comes back to what Chris was explaining about python being the interpreted language and when you're running python code you're actually running The Interpreter so what they've done now for python 312 and again this has been pabl gindo alato has been involved in this and they've added support for perf by using a technique called trampolining or trampoline instrumentation and mainly it's just a small hook that you need to activate and then it will expose the python calls to perf so that when you set everything up then you you can actually use this profiler and you'll see things like okay it's it's spending 90% of the time in in this function so that's might be where you want to go look for for optimizations and so on yeah as I said this is something that may be just what you need or it's may not be that interesting uh but again if if you want to learn more about this we have a fantastic Deep dive article that BTO zinski wrote so so he he Dives deep into how to set up the Linux perf profiler which is not completely trivial so it's it's already good resource just for getting this thing up and running uh then it shows how you can run it on different Python scripts ways that you can kind of yeah look at different functions set up some Benchmark scripts and then also he goes through several tools that you can use to look at the reports that you get um get from these uh profiling runs and to top it off we also added an article about profiling in general showing uh some of these other tools that we mentioned earlier like the C profile and time it and so on yeah so there's lots of stuff to dig into there yeah in fact um I think after that article came out that bartage did about General profilers that's when uh emry Berger reached out to me to talk about scaling right yeah he he said bartage us should check this out and then he referred him to me and so we had that episode which was very enlightening if you want to do a much deeper dive into profiling and what's uh capable of some of these more advanced uh tools for seeing how your code runs and maybe finding where bottlenecks are Christopher you want to talk about Immortal objects sure we touched on this not that long ago we talked about it in episode 171 yeah but it's a 312 change and so I'll go over it quickly just so you know for completion sake he says knowing he never does anything quickly so cpython uses a internal structure that corresponds to every object in your python code and of course everything in Python is an object I'm going to get a t-shirt printed so maybe I should just have I should have a bell I can ring on the show uh so so this uh this C python structure includes both the data for the object which kind of makes sense but it also has metadata about the object and one thing that that metadata is used for is tracking how many other objects are referencing your object and that gets used to know whether or not if nobody's using it anymore we can garbage collect it so so that's the first pit the second bit is objects that don't change we're going to call those immutable immutable objects can still go out of scope which means immutable objects have to have mutable metadata because the metadata has to continue counting whether or not this thing can get removed or not so enter Immortal objects pep 683 identifies a subset of IM able objects that actually typically last for the lifetime of The Interpreter and one of those is the nun object CU even none is an object in Python and so an immutable object that lives for the lifetime of The Interpreter that mutable metadata section means you're never going to garbage collect it so there's this subset of objects that live forever and they've got this extra overhead so what essentially is being done here is a flag is being added to identify them as not just immutable but Immortal and then they'll never need to be garbage collected ever and that means some of the metadata can be optimized away you don't have to worry about things like cash invalidation you don't have to worry about synchronizing them across processes because the two processes aren't going to change them so you don't have to sync that and the removal of some of that metadata saves memory so this pep came out of the folks at Instagram they're a Jango shop and they've seen significant improvements to their startup times and memory usage just by implementing this optimization so long live Immortal objects completely internal change again unless you're writing an extension you will never see this but they're at scale this is a performance Improvement be aware certain objects are Gods yes yeah yeah there there's a very dated Buffy the Vampire Slayer reference in there but I suspect it's too so I'm just going to leave it alone okay for this week's video course Spotlight I want to remind you that Christopher Trudeau has created a video course all about this week's Topic in this video course you'll learn about the new features and improvements in Python 312 improvements to FST strings more did you mean error messages new functionality in the standard Library including features for pathlib and Inter tools additions to static typing and changes in The cpython Interpreter that continue the work to speed up python if you'd like a guided video tour of the new features in Python 312 this is the course for you like all the video courses on real python the course is broken into easily consumable sections plus you get additional resources and code examples for the technique shown all our course lessons have a transcript including Clos captions check out the video course you can find a link in the show notes or you can find it using the Search tool on real python. gar you want to talk a little bit about some changes with the calendar uh right so this is a fairly small addition that they're doing to the calendar module and I must admit that one of the reasons I want to highlight it is to just highlight that python has a calendar module which may not yeah be that from everyone knows about uh so so if you're working with date in Python you're probably using the datetime module which is kind of what provides date and datetime classes that you can use to represent well dates and dates with time stamps in the calendar module they' kind of sort like adding stuff that is more towards say say the public use in a sense the less technical use of of dates and so it can just show a calendar if you if you run the python minus M calendar that's how you run a module it will actually just print the calendar in in your terminal so so there's some NE neat functions there and then then they have some um some functions for just calculating days of the week they have the list of of months in in your local so in your language and things like this are inside of this calendar module the one little thing they're adding now for python 312 is that they're adding constants for each of the weekdays Monday Tuesday Wednesday and so on and for the months and these are enumerations which is essentially a collection of constants and they they're represented by numbers as well so you can kind of if you want to you can both refer to say the 10th month and you'll get October back you can refer to October and and kind of get the number 10 back or you can just iterate over all the months and things like this so it's just making the calendar modules slightly more useful by adding these not a big not the big deal at all but it's it's a fun little addition to the language this one really surprised me I don't understand how we've gotten to python 312 without having this already sort like what do you mean we didn't have this all right get to there get it eventually whatever that's fine so yeah beforehand you would have been importing something potentially to help you with this I think there were third party tools that did it I suspect I've got code where I've done it by hand myself yeah yeah and they had uh they they had like the months available in in a list and things like this before but okay the the constants are kind of the new stuff yeah nice or the enumeration I think even the week T they had as con constants but not related to each other all right well maybe that we remove some imports there potentially for some people down the road speaking of which this is a one I I really enjoyed your coverage of this in the in the video course Christopher the the thing about batched in itter tools yeah it's uh it's a nice little again it's it's a small little Improvement but it's one of those that's kind of useful you never used it there's a third-party tool called more itter tools you should check it out it's got all the kinds of things that you kind of think should be an itter tools but don't exist and one of them is something called chunk and chunk takes an itable and returns some values grouped together so this is really handy if you want to take a bunch of data and present it in say rows you chunk based on the number of columns you want I've used it a bunch for writing things like that in the web and now because of python 312 you no longer need it for that because the same concept has been built into the actual iter tools Library they've named it batched instead of chunked but it essentially does the same thing so let's take the new calendar.on enum and say you wanted to print out the month by quarters you could call it. batch passing in the calendar. Monon month enum and a value of three uh which is the size of the group and you'd get back four topples one for each quarter and each the Tuple would then contain the three group months it's handy for say printing it out to the screen or whatever and you know a formatting that kind of stuff so useful little thing yeah I could really see how it be handy you know with layout stuff uh for D Jango and whatever i' yeah I've used I've used chunked a lot on uh in order to do rows and Columns of you know like hey here's your video live liary you don't just put a straight listing of videos you put three or four columns wide if you're using you know bootstrap underneath and you've you've got that concept of rows and columns inside a bootstrap you're almost always doing something in your view in order to format it in order for bootstrap to be able to process it that way so yeah this is useful little functionality and it may for me at least it's going to remove one extra dependency because it'll be built in nice so Gana we got to talk about types again seems every every release we got to have some new stuff which is great in fact uh it looks like they're simplifying some stuff which has always actually been a bit of a theme in some ways um making maybe the types uh look a little bit little bit cleaner as we go through it is that the case this time too uh yes definitely and yes you hint that so that there's always new type improvements in Python it seems like so the whole support for static typing was added back in 35 and then they've kind of gradually improved it and I think one of the guiding principles is that they kind of want to try things out before committing to them fully oh yeah so so it feels like the we can kind of broadly categorize the new type changes into two two things one is kind of we're getting new features and those are typically added into this typing standard Library where you can or you need to import stuff to try it out and and so on and it's I guess the idea is that it's easier to remove things from there um in addition um stuff that's in the library you can also backport uh so they're using a third party Library called typing extensions okay where where you can put features so that you can use them in even in earlier python versions so even though you add a feature in Python 312 to typing if you're on python 39 you can import that same feature from typing extensions so that you can actually start using it earlier or on older pythons then the second kind of category of of new things in in typing uh are when they're changing the syntax so they're actually changing the Python language to support the typing stuff and some examples where they did this was when I think that was back in 39 we could start using lowercase list and lowercase dict to annotate types or list and dictionaries uh before 39 we need to import capital or upper uppercase list and upper case dictionary to to do the same kind of annotations and then in 310 they added specialized Syntax for doing unions and so on so in 312 they are simplifying and adding dedicated Syntax for type variables this the type variables then we're kind of getting into somewhat Advanced type territory but uh I think the the new syntax makes a lot of sense when you just get used to it so to go back a little bit a type variable is uh a variable that can represent uh different types during type checking so for concrete example say that you have a function that can work with a list of elements and uh often that you don't care too much about what type of elements those are your function can potentially work with all of them so just to do a very simple example of this you may have a function that can pull out the first element of a of a list and then if you want to add a type annotation to that you want to add State what's the return type of this well the return type would be whatever is the type of your list elements and mostly when you're working with lists you have some kind of homogeneous list you have a list of numbers or you have a list of strings or you have a list of booleans or things like this so what you want is to be able to say that okay I'll I'll my function can work for any kind of list but if you can identify that it's a list of integers then I know the return type of my first function will be an integer or if it's a list of strings then the return type will be a string right so one way to do this would just be to write out many many type hints for all the different types but it's much better to to use a type variable so what you'll do then is that you'll just declare a variable so up until 312 you needed to import something called typear from typing and then you need to declare that say t is a commonly used variable for this but it could really be anything but but say t equals a Type R and then inside your type end you can then say that elements is of type list of T and then my function returns T so then the T will stand in for whatever function of whatever type that your list happens to be uh so then it can type check if it knows that okay now you're passing a list of integers into first the type Checker knows that it returns an integer and so on really this type variable is uh just something that can stand in for in general any type and then there are ways you can constrain it to okay this function actually just works for different kind of number types so integer floats and complex for instance or something okay uh so up until now as I said you need to import this type bar you need to declare it beforehand and so on H in Python 3 12 they introduce a new syntax which looks a little bit weird the first time you see it but you're now allowed to write in your function definition you'll write something like Def and then first and then before the parenthesis where you have your parameters you'll add a square bracket and then just declare your type variable so it will be the first square bracket T and then you end that square bracket and then you can list your parameters afterwards and just by putting the T in there you're you're saying that okay this function is what's called a generic function it can work on different types and this T is a t type variable that I want to use inside of my Ty pins in the function yeah the first time you see it there's kind of okay what is the square bracket doing there but I think my feeling is at least that uh this is nicer than trying to figure out all the typew stuff before it kind of looks it looks fairly clean actually and you can use the same syntax when you're defining a class if you have a class that's kind of depending on on the type in the same sense the generic class as they called okay in that case you'll do the class and then the name of your class so for instance if you have a stack for instance you can do class stack and then square brackets t or whatever type variable name you want to use and so on these type variables as I briefly mentioned they can be constrained to only be of certain types so if you want to constrain your type variable then inside of that square brackets where you declare it you just add a literal Tipple of types so you can write something like T colon and then parentheses since it's a tle int comma float comma complex for instance then you say that okay this is type variable but it can only be either an in float or complex it can't be any other type okay and the final thing it can do is that it can also be bounded which means that it can be of a certain type or any of its subtypes or sub classes so you could say something like this thing is bound bound it to be a string or any subass of string the Syntax for D is also using a colon so that will just be T colon and then the type that you're bounding it to so T colon for inst for Strings the final place where you can use this type variable syntax is for type aliases um so type Alias is also something that's been supported since the beginning type Alias is just a different name for a type often something you'd use if you have a some kind of nested or complex type so if you have say a a list of tupal that are say a a tuple of a callable and an integer or something like this it it gets both hard to keep typing it out it gets hard to read it kind of gets typically you you get lost in technical details of what what is this type really and then maybe it is just really I don't know some list of plugins or something like this okay uh so so then you could just create a new type that says this is a list of plugins so these type elas can also be generic in the sense that they can depend on these type variables for type variables you're now allowed to start using this t as well but since that doesn't really yeah so far a type LS has just been a regular assignment so you just say this list of plugins equals and then the type but then that's not valid python syntax to suddenly throw in a square bracket on that so they added a new sort of like statement where you write type in front of your your new variable name and that creat a type Alias so you'll write something like type space list of plugins equals then whatever that type is so so this is actually a new soft keyword as they call it so type already exists in Python uh as something you can it's it's a buil-in that you can call to just check the type of of of of an object yeah so here they given is a second use essentially and by using what they call a soft keyword it means that it's just a keyword in certain contexts so it's a keyword when you're putting it in front of a variable uh but then you're still allowed to use the the type built in where anywhere else so so again it's just uh simplifying and making more explicit syntax uh for this especially the the generic type alas is where you can use the type variable um it looks much neater now than it used to I think it will help projects that have to accept lots of different type types projects like that do testing and things like that for them to be able to add this sort of functionality hopefully it'll make their code look a little cleaner with the annotations it and right it I think yeah whenever you kind of end up with your own kind of containers so your list like structures these kind of things are super helpful cool and yeah I can note for for this one as well we have a preview article that kind of goes more in depth than we can do in the in the main article and that's one that I wrote myself yeah that one just came out also right before this yeah y great Christopher you have an update of some new feature for pathlib I think I've mentioned on the show before that uh I'm a fan of pathlib uh it's been around a while but somehow I was a late adopter of it if you're fortunate enough to be fully in the Python 3 world you really should be using it instead of os. path for all your file name needs common thing you do with a directory is to figure out what's in the directory there is a method in pathlib for iterating on a directory but it doesn't descend it only looks at the level of the path so historically if you wanted to descend the file tree you write a little recursive function and then you know basically call yourself with this iterator so that always felt like a weird bit of a shortcoming because the OS module actually has a walk method that descends for you well now 312 so does pathlib right so you can walk a path object that is a directory and you get back a series of tuples very very similar to how osok did it each tupple contains another path object for the directory inside of it and then uh so would be you know the first one you're walking and then the second item is a list of the names of subdirectories and then the third item is a list of the names of file names inside of it so you essentially can see a whole picture uh and then the call to walk desc