Measuring Multiple Facets of Python Performance With Scalene | Real Python Podcast #172

Key Takeaways

welcome to the real python podcast this is episode 172 when choosing a tool for profiling python code performance should it focus on the CPU GPU memory or individual lines of code what if it looked at all those factors and didn't alter code performance while measuring it this week on the show we talk about scaling with emry berer professor of computer science at the University of Massachusetts Amherst emry talks about his background and memory management and his collaboration on horde a scalable memory manager system used in Mac OS we discussed the need for improving code performance on modern computer architecture he highlights this idea by contrasting The Familiar limitations of Moore's law with the Lesser known rule of dinard scaling working with his students in the University lab they developed scaling scaling is a high performance CPU GPU and memory profiler it can look at code from the individual function or line by line level and Compares time spent in python versus code emry talks about the recent scaling feature of AI powered optimization proposals and covers a couple of examples he also shares a collection of additional python code assistant tools from their lab all right let's get [Music] started [Music] the real python podcast is a weekly conversation about using python in the real world my name is Christopher Bailey your host each week we feature interviews with experts in the community and discussions about the topics articles and courses found at realp python. after the podcast join us and learn real world python skills with the community of experts at real python. so I want to welcome emry Burger to the show he's a professor in the College of information and Computer Sciences at the University of Massachusetts Amherst and welcome to the program oh thank you great to be here real quick it's question on the College of information and Computer Sciences how does information kind of evolve in there I I'm wondering how that is adapted in some ways if that's kind of a newer distinction for colleges yeah so it's a a weird situation it's kind of a historical artifact but it's also part of this program that we have so we have a program called informatics okay which is really more broad I think than traditional computer science but the brief history of this is we started out as a department and then we became a school of computer science and then we went to become a college which is the highest level um hierarchically in a university and the Provost at the time was like well you need to incorporate information like an information school or something like this and that's the actual cause that that led to that that name I was actually it was originally meant to be computer information and science or something like this and I I was at a faculty meeting and I was like if the words computer science are not in this college then I am going to go move to a place that still has computer science in the name so so they kept the name which I was happy about that's good it sounds like you have a lot of background in memory management systems one of the things that you've listed on your blog is you worked on a system called horde yeah and I was wondering about this background and memory management systems and does it help when I mean we're here to talk about scaling which is a I'll put your your words a state-of-the-art CPU plus GPU and memory profiler for Python and it has now ai powered optimizations and suggestions but does that background in memory management systems does that help you working on a profiler yeah I I mean I certainly think so so the hord project that you refer to it's something I I worked on during my PhD and afterwards and the algorithms that are in horde got significant uptake you can actually find a comment in the Mac OS implementation of Malik that cites our our hord paper which is pretty cool yeah so you know memory management is a an area where you have to be aware of lots of details of what's going on in the hardware uh what's going on in the software you know performance really matters a lot for memory management because it's uh so heavily used right like you're constantly creating and destroying objects and doing all that performance engineering required really uh understanding what's going on in the system and being able to follow what's going on so I was already a client of profilers at the time because you know there's just no way to know you write some code and it runs a little faster a little slower and you're like well why what happened and some of the techniques that we use used for building replacement memory allocators are actually in scalen like I'm using a library in scalen that uh I started working on for my PhD for writing memory allocators so that's actually that that code is actually in scaling oh that's cool you you already kind of mentioned the idea of or at least the theme of several of your talks recently which all include links for about performance mattering and then uh recently injected really matters and uh also another key word in there is again which I think is kind of interesting um in these talks you you talk about how the architecture of computers have over time hit walls and I thought it was interesting that a lot of people have a confusion about sort of Moore's law versus uh you mentioned dinard scaling and maybe you could discuss that real quick just like what's the difference going on there and why people might maybe confuse one with the other yeah I so that's you know this Mor law dard scaling thing everybody's heard of Mo's law yeah very few people have heard of dard scaling but dard scaling is really what made our computers faster for all of these years not Mo's law per se so Moors law was formulated by Gordon Moore of Intel yeah who said basically it appears that roughly every 18 months the feature sizes of semiconductors that is to say the the size of transistors were getting half as big every 18 months right so you could put double the number of transistors on a chip every 18 months and as this was happening basically this other phenomenon called dinard scaling was happening which was that you could increase the the Cycles the you know the megahertz or gigahertz yeah proportional to this this increase in density and basically the intuition was if these things are denser this means that the uh well let's just suffice it to say that the the shorter the wires are if you will wires the little etched tiny things on these chips the faster that the you know the communication can happen because speed of light right it's actually traveling L distance and dard scaling basically said well you can actually also increase the frequency okay so we were increasing the frequency along with increasing the density and that lasted a while and it was it was great and I you know I think so I recently relatively recently got an one of the Apple m1's okay uh and now of course they have the m2s and when you shift from the Intel technology to M1 sorry everybody from Intel out there uh but when you shifted to the M1 all of a sudden the computer was much faster like everything ran much faster and that was the first time I had experienced that in 20 years and you know it used to be back in the day from you know the late 80s or so on chips were just getting faster and faster every year and you would buy a new computer yeah you know youd buy a new computer two years after you B the previous one and would run more than twice as fast yeah and uh and so yeah it was it was a time when people openly were derisive of of optimizing code and uh caring about performance because they're just like just wait just wait just buy by next year's computer yeah this hard will run it faster yeah yeah so somebody actually I talked to somebody once in Texas so I was a PhD student in Austin and there was somebody at this supercomputing Center and it was a a weird dilemma because it was like well we can buy a computer now like we place an order for it and then eventually it gets installed and by the time it's installed it's obsolete yeah and the new thing is like 2x faster and you know how what is the likelihood that you programmers are going to get out there and optimize things and make them two times faster in 18 months versus the hardware which just does it magically right and and so it's kind of a you know for them it was a dilemma but really it's a great situation to be in because you know this Rising tide was lifting all boats but uh but that is really not the case anymore I mean Apple has done amazing things and you know we could talk about gpus or something which obviously have gotten faster and faster right but you know that's mostly through parallelism on gpus and with the M1 and M2 it's mostly because of system on chip and much much larger Hardware caches and stuff yeah keeping the memory so close to the to the processor and all that yeah it's making those are the interesting speed s that are happening but generally like overall processing it's you can compare almost like Raspberry Pi products to some other you know computers today and it's like well it's not that difference in the base amount of like Cycles this thing is going to run which is kind of shocking yeah yeah because you know if you looked at the curves you know we were supposed to be at the terahertz range by now uh and we're still you know about three or so gahz and you can't really get much faster without you know like liquid nitrogen on your motherboard kind of situation yeah yeah yeah though I'm hearing interesting things about uh superconductors in the last couple weeks so yes we'll see what happens with that we will see we will see yeah yeah so when did you start was some of this performance something that you were thinking about at the beginning of Designing a profiler um was this something that kind of tied into it and maybe you could talk about the background of like where did this project start up and and how long has it been going sure so I mean I have been working on performance related tools really just uh developer tools for most of my career we have a profiler not for python uh but for native code like C C++ or rust called cause coz uh which is a very cool fun kind of profiler that I'd be happy to talk about it doesn't have that much to do with scaling though but yeah the python profiling thing actually came from my own suffering with performance issues in Python so what happened was so I have this website called CS rankings. org which is essentially it's a metric-based ranking or portal to computer scientists in Academia and it's intended for grad students to be able to select what kind of topics they're interested in and then it shows them which faculty at which institutions are working on those topics and then they can drill down and click on their homepages and do all this stuff so to make that scale I made it a static website and it's hosted on GitHub pages and this means that there's some processing that has to happen of this database that I don't want to run server so I basically just run it on my own computer and it rebuilds this database and generates a bunch of csvs that are easily loaded by the JavaScript that's in CS rankings but that process was taking ages uh it was taking just a crazy amount of time how often was it running I would just run it periodically okay because it was so painful it' be like well don't use your computer for 20 minutes okay and it's a python script and you know the python script was getting bigger and bigger and the first thing that I did was I said well let me just run it with C profile and then C profile was really totally useless because it just listed function level information and you know yeah I had one big function that was doing most of the work and it was like it was it was yeah it's a m oh you're sping time in main buddy yeah it was really disappointing and then I went and looked for line profilers and those weren't very good and then I was like oh but it consumes a ton of memory let me run memory profilers all those csvs right yeah it's processing this gigantic XML file okay um I mean really gigantic like I had to write a separate Java thing to parse the XML and essentially down sample it to reduce it in size because it was so freaking huge wow so and then anyway there's it's kind of a mess but be that as it may I was like wow there's like no good profilers out here I was really surprised by this and memory profiler in particular it slows down your program by 100x a thousand X and so you know take 20 minutes and multiply by a thousand yeah like I don't have time for that so so then I was like wow I I guess we should be looking at this and I was surprised to see how kind of traditional the existing profilers were they didn't look very different from the profilers that were created for C in the 80s and you know early 90s it was basically GPR okay and I was like well this is how how are we here like this is C is not python right and Python's doing all this very different stuff especially with data today these large chunks so there's like lots of unique ways to look at it and also you know we've talked offline before we started just that everything is in the data science world uh or anywhere people are looking for Speed all is loading into C libraries and other things like that abut what's looking at that so yeah yeah so you know it would just tell you hey you're spending a lot of time somewhere and you're like well okay but is that like is that time I have to spend or is there some alternative and so one of the first things I wanted to know was what is the what's the breakdown here like what are the things I can do like one of the things that really I think this is a problem we talked about this offline before as well with uh with profile is that you know a lot of these profilers have a bunch of information but like a lot of this information is not useful to me like you know uh you could this is not a thing profilers do but imagine if a profiler said hey you've got a lot of if statements that you're executing right great I'll definitely take out if statements right it's it's not super helpful right so you know what I really wanted was a profiler that would tell me where the problems were give me really good advice on how to fix them and make the make the thing run faster or consume less memory so that kind of helped you develop a set of targets that you were going to sort of attack and say you know what are the Realms that we could look into and obviously you just said if it's just going to look at functions that's probably not going to solve it for me so where where did you end up dividing it up then yeah yeah so the very first thing was all right I want to do a statistical profiler uh that was just periodically sample where we were I Tri to implement one using the buil-in facilities for using profilers but I just found which is what C profile does and I just found it distorted the execution time a lot and I just didn't trust the results and I measured some of the times and I was like this this is just off uh so I I went to use a statistical profiler and the the idea of a statistical profiler is very very simple you basically periodically you set a timer timer goes off you wake up and you say oh what code am I running right now and you just do this over and over and over again okay and the ones that you see all the time that's obviously where you're spending all your time so it's kind of like a like a track meet and somebody's actually looking at lap times and trying to figure out performance you know as the things going not just the overall end yeah yeah I mean you could imagine it's a great analogy I I love this analogy I'm gonna have to steal it sure yeah so you know you could imagine if you had uh every person on you know who's at a track meet had to actually carry something like some big timing device like a giant Flav flave clock on their chest right and they're running and then you know this is keeping track of their their instantaneous time all the time okay this step took this long this step took this long um it's obviously slowing them down right it's distorting the actual their actual race is is no longer the way they would have run if they weren't carrying this giant thing but but instead if you just take a photograph periodically right of where they are you know who's ahead right you you get a really good picture if you have enough photographs you get a very good picture with really no overhead right without distorting the the underlying results yeah okay so the other thing I really wanted to do right up front was to also Trace memory so this is part of the like memory allocation background I was like well you know I don't first I don't want it to take a thousand times longer that's crazy and second if you just tell me my code took a bunch of consumed a bunch of memory that's like a dimension and then there's how much time it took that's another dimension and I wanted everything together I wanted a big picture of what the code was doing not just in terms of runtime but also in terms of space that makes sense the memory profiler typical one that would run in you know like part of the Python tools that a lot of people are familiar with was looking at every single allocation and basically documenting that in a way like kind of like keeping track of every single allocation so I could see that again goes back to the analogy we were talking about of just like um now I have all this overhead of like you know of tracking how many thousands or millions of potential objects are being created and allocated and and removed and so forth so it kind of makes more sense to me that the idea of what you're doing with the memory profiling that's happening inside of here in fact you kind of set thresholds I think in scaling where it it's only really paying attention to like it bumping up against different sort of levels as it goes which I I found kind of fascinating too I thought of the term quantizing as you know because I'm into music and and timing of like sequences of stuff and I think same kind of thing it's like okay I'm just seeing which makes sense you know from a sampling perspective like it's enough information to really see where the problems are right again could be oh there's a photograph for the where they're stumbling as they're running along I can kind of start to see where the problems are it's interesting that that doesn't take very much overhead at all like it there were a couple of different kind of factors that kind of went into deciding that I me I remember you mentioning that it was around a megabyte or something that it was looking at and then you said it was in one of your talks uh actually like a prime number and I don't know if that's too convoluted of a thing to get into yeah I mean the prime number is just a heuristic okay to try to avoid some sort of like if you if you're allocating things and you're con that the intuition is just well this prime number is a very weird number in a sense and it doesn't like okay things don't divide into that prime number very well so if you're allocating chunks of memory the chunks of memory that you're allocating are almost certainly not going to be exactly that prime number and then they're not going to divide evenly so you're going to end up with it kind of skewed as you go in the sample but but really that particular approach is not anywhere near as important as the kind of the growth or shrinking threshold yeah yeah which which actually is the thing that matters I mean it's sort of like okay so first let me just say Python allocates and freeze memory like crazy like it itself is just a giant allocation machine well it's everything's an object that's part of exactly it's creating objects all the time and and reclaiming them and so on so you know doing something that is actually just tracking every individual allocation is just going to occur a ton of overhead just by by the nature of python uh and you know people have done memory profilers for you know for C C++ and you know you want to keep those low overhead as well but the allocation rates for python are an order of magnitude higher than for most C or C++ programs you know in see an INT is an INT it's not an object you allocate arrays and you maybe even do more sophisticated things that are just Out Of Reach for python yeah so you know essentially what we're doing is we're just trying to keep track of how high the memory consumption is so we don't really care oh it's allocating and freeing a lot yeah of course it is it's python it's allocating and freeing like crazy but if it's sticking within kind of a narrow bound of less than a megabyte then we don't care but if it's growing every sort of one you can think of it as a mile mile marker right every mile will take we'll say oh you went up one two 3 four and so on and so that's really the Insight the Insight is yeah I mean it's it would be expensive we kept track of it all the time so we just don't yeah the only thing we do is we keep track of how high we have gone or how low we've gone since the last time yeah you call them the like kind of high water marks and indicating them as you went and then I thought it was fascinating because like the graphs that the program outputs scaling outputs are actually pretty interesting there's an HTML out put that you could move your mouse along and and kind of see the behavior and so if you have a lot of data moving in this sort of memory allocation happening this huge amount of chunks and it becomes like you said like a Sawtooth pattern where it's like large jump up and then all of it drops back down that's that's definitely a performance hit especially you know when you think about like just the speed of memory versus like something that's already in the processor and already working so where does somebody look if they see that kind of error like where do they go look on their program yeah so in the visualization the visualization that particular one that you're talking about is kind of memory use over time yeah right so it's it's like oh it's going up it's going down it's going up it's going down and you know if you see something that is weirdly producing a bunch of kind of whip sawing right that's going up and down up and down up and down by a large amount that's a little troubling it seems unlikely that you intended to do that right and so one of the the cases that we've seen where people do this there's sort of two things that are pretty big culprits for this kind of behavior one is when you make a move accidentally between native representations like a npire array and python array okay and you can accidentally do this by like you know using python indexes or comprehension or something like this and then it'll be like well uh I need to turn this into a python data structure so it's to allocate a ton of memory and it does whatever python thing and then it throws that thing away right right the other thing that happened that we found was just a totally random numpy thing which is just getting into the weeds for a second about numpy numpy has this thing everybody uses which is np. array and np. array takes a python array or Matrix and converts it into a numpy one so numpy can operate on it and uh somebody sent us a piece of code as a bug report actually they're like oh it's producing this behavior and I it shouldn't be and it turned out that what they were doing was they were calling np. array on an already numed array okay so uh it turns out that uh when you do this numpy actually helpfully makes a copy of the array all right let me make you another one yes hey you like arrays I hear you like arrays here's some more arrays yeah yeah and so it seems totally innocuous you would think it wouldn't do that but that's actually the default behavior and it's just one of those weird pitfalls that I had no idea about until we saw this problem yeah that's cool to kind of find these one of the things that scalen is doing that is interesting is the the line by line analysis was that helpful in that circumstance oh yeah yeah I mean this it was specifically saying here's the line here's the line where this is happening okay I mean one of the the big differences between like C and Python and there are many but one of the big ones from a profiling perspective is that in Python just a lot happens on a line yeah right you can have a line that's doing a ton of stuff right you can be like one line of code oh it opens up a web page and it downloads some some content and does a transform on it totally normal in Python right you would never see this in C okay right this is just not C would be like there' be a load command and there'd be something else and something else it's it's always much more granular in C uh and so having line level profiling in in C can be useful but it's really when you're chasing Cycles you're like oh I need to you know get this inner loop down to like 20 Cycles or something because everything is so fine grain but in Python it's usually just doing tons and tons of work yeah it's yeah it's kind of fascinating like how many things are are called you know especially if you're calling libraries and methods from them it's like you're not even sure like what overhead is happening inside of that yeah yeah which which is you know which is great I mean this is one of the great things about python right it's so convenient to do these things but um there's an old saying about Lis and Fortran which is that basically something to the effect of Fortran programmers know the cost of everything and the value of nothing and list programmers know the value of everything and the cost of nothing and so python is kind of similar in that you know you have access to all of these things you can just have one line of code that one line of code could be responsible for doing tons and tons of work and you just don't know AR priori how much time it's going to take or how much memory it's going to cons yeah you don't know what the expense is exactly that's pretty wild this week I want to shine the spotlight on another real python video course as you started on your python Journey you probably encountered a common chunk of python code and wondered about it or maybe you even included it in your code and wondered if you were using it correctly well this course answers the question right in its title what does if Dunder name equals Dunder main mean in Python based on a real Python tutorial by previous guest and real python core team member Martin Bryce this video course is presented by Ariana D and she takes you through not only what does the dunder name equals Dunder M idiom mean but also how it works and when you should use it and maybe when it's not the best fit whether you run code as a script or you're importing as a module this course will answer your questions on the topic real python video courses are broken into easily consumable sections and where needed include code examples for the technique shown all lessons have a transcript including closed captions check out the video course you can find a link in the show notes or you can find it using the Search tool on real python. one of the things that you you were doing along in this process I'm not sure where this came in I just again having watched your presentations that you did at Pyon later that you showed a chart of other profilers which I thought was interesting and I was like well how did you measure the accuracy of all these different profilers how how did you get those widely different results in that some of it you've sort of explained like the ones that are doing that memory allocation type of thing are definitely weighing down the program a lot because it's sort of sitting on top of it but were there other things that you were finding as you were and how did you do this measuring of accuracy yeah so you know we we were just curious where like well we assert that statistical profiling is better okay I think you can kind of reason it out that it makes sense like the way I described it but we're like well how much of a difference does it make in practice so we wrote a pretty straightforward Benchmark the the problem with profiling and measuring accuracy of profiling is that normally to find out where a program is spending all of its time you use a profiler right so then you have to be like well this is the you know this is the golden one true profiler somehow and everything else is different and that's not very satisfying because you know if yours is the one that you've just decided is the golden one well all the others are different and you're the winner you know hoay for you so we needed something that provided ground truth right so we wanted to be able to know look it should say it spent this much time in this function and this much time in this code and so on so we just constructed a micro Benchmark that would allow us to do this and the way we did it was really simple we made these two functions or these two chunks of code take different amount of time and we just measured it from the beginning to the end of these computations using the highres timer okay and so so this allowed us as long as it's you know the amount of work that you do is much larger than the cost of actually talking to the timer and getting that value then this is a totally accurate way of measuring this this cost so we went and we you know basically ran it through a gamut of different proportions and then said well you know what did these other profilers report and so the profilers that are the least accurate are unfortunately the most widely used so cprofile is one of the least accurate and the reason is basically because of exactly this situation of checking every single thing as it goes you know it uses the built-in facility that python has called set profile and set profile there's actually some a totally improved way of doing profiling I should add that that you can do starting with 3.12 but even so doing it with this kind of Trace based approach where you're actually collecting the information as it goes it just unavoidably distorts things okay uh and you really want to do it in this much less intrusive way of of doing statistics where you just take these snapshots that makes sense yeah like it seems that there are these kind of tradeoffs of like running additional code as opposed to sort of just doing measurements that that are going to cause that one of the things I wondered about though is like in a similar way how can you look at something like memory leak detection is that something you can still do uh through a form of just sort of monitoring and looking at what's happening as opposed to you know trying to pay attention to everything so how did you come up with your ways of determining memory Leak Detection yeah so the memory leak detector is yet another sort of of let's try to do this with sampling okay so the idea behind the memory leak detector is again I you know monitoring every individual thing is very costly another thing that memory Leak Detectors do is they save all of the state of the Heap like you take a big snapshot and you compare the snapshots this is problematic actually in Python because there's python based leaks there's native code leaks there's python plus native code leaks there's just a lot going on and so it's and and it's you know traveling through these heaps is not really a great way to spend your your day let's just say you know I really wanted a tool that could just automatically finally expect they could be pretty large yeah it's it's not fun you're like hey these two things should be roughly the same let's compare all the Heap data structures and do a keep diff and it's I've done it before it's really not productive sure yeah so the idea behind it is pretty simple but but it took a while to come up with it the idea is that what we do every now and then is we just say hey you allocated an object I'm going to keep an eye on that object and let's see if I later free that object okay right and so I just have some period of time and I'm just like oh I grabbed one like it's like a stream of objects going by and I just grab one so I grab that one object and then sometime later I look to see I actually have to check every time I call free but basically uh the it goes oh I'm going to reclaim an object reclaim an object reclaim object and I just check is it one of the ones I sampled okay all right if it is then I say oh good it was reclaimed it's not a leak probably all right and so I'm kind of accumulating evidence about all of these allocations right so I allocated this object from line 12 and whenever I allocate an object I eventually see the Reclamation of that that would if that happens enough I'm pretty much going to assume that's not a Le okay but by contrast if I I I say I sample an object from line 12 uh and some time goes by and I'm like well time to sample another object that one was never reclaimed while I was while I was watching it could be a leak so I don't know if it's a leak it could get freed right after I do that right the next time I take a sample could be like oh yeah I now I freed it right so it could be a false positive but you know we expect this to be kind of a random process so we just say oh maybe it's a leak and then if we keep seeing maybe it's a leak maybe it's a leak maybe it's a leak we build up confidence that oh it really probably is a leak it's like a interesting QA system where it's just sort of like like again doing the sampling and kind of measuring and it's like well typically things happen around here and this is being atypical and so that starts you know start building up the evidence yeah to say all right this is needs to be investigated right right and you know one of the things that you you know you might want to do is to make the uh the way that you take these samples proportional to the size of the objects because if you have an object if you have a little tiny object and it's leaking very very slowly we kind of don't care okay right but if you have a big object then you know that thing doesn't take much to leak before it becomes a very serious leak so we measure this we call it leak volume uh and whenever we come up with something that looks like a leak we report to the programmer hey the leak volume was this and if you see it's like you know four bytes a year you're like okay I don't care about that but if it's you know three megabytes a second you might be concerned we're headed for trouble our raft is leaking yes so I thought we could kind of dig into a little bit of somewhat use cases but also like we mentioned some of the different ways this tools working but I always kind of wonder about where in place does somebody look at hey I should start doing this we mentioned performance and a lot of the things that come up are definitely this use of python for data science these large amounts of memory being used these these collections of other libraries that we mentioned that you might want to have again evidence of like hey what's happening why is this maybe performing poorly but when does somebody look at like if you're going to generally say hey when should you look at using a profiler you know what what is your typical use cases and when do you introduce it yeah so I mean I think it does vary but you know often it's just like oh this code is annoyingly slow okay right so you know I'm in Jupiter and I'm I'm doing something interactive and I run something and I I get I'm sitting around waiting for a result yeah right so once performance becomes a paino that's usually the right time to to grab for a profiler I mean the thing you definitely don't want to do is to go on your gut instinct and say oh I know why it's slow because you probably don't right but but you know what does know a profiler knows a profiler knows you know what's slow what's consuming memory at which often has the same same knock on impact of slowing things down one of the things I haven't mentioned about scaling which is really distinguishing is that it doesn't just break say oh this code took this much time it actually breaks it down and says it took this much time running native code like libraries it took this much time in Python it took this much time in the system like executing IO so it gives you this information you know there are things like if it's like oh it's running native code it's not consuming IO uh like I'm already doing the right thing there right even though it's spending some amount of time that is not really a candidate for optimization yeah but if you see something it's spending all of its time in python or all of its time waiting on iio you might think well if it's in Python I need to re rewrite it to use libraries if it's waiting for Io you might want to start using a syn of weit to try to hide that latency things like that okay so that's that's one case just the like oh it's slow this hurts me but you know the other case the other case is like you know uh if you're running stuff on the cloud for example the amount of memory you consume dictates what kind of a uh computer that you're going to actually end up renting sure and more memory is more money right right so so you know more slow more RAM means more money okay and so you know if you're if you built a system and you're like well I really would like to use a smaller system or a less you know a one that's really shared and has it's just cheaper then doing some profiling will probably help you find places you can cut cost is there anything that somebody needs to keep in mind if they're looking at running scaling on a cloud instance and wanting to measure its performance are there things that somebody needs to keep in mind in installing that or is there anything unique about that um so the only thing so we recently added this actually specifically to to address the concerns of folks using it so so scaling is mostly an interactive thing you say scaling and then you you know you run you write the rest of your arguments you run it and it produces this web based user interface but if you're running something in the cloud right there's no browser in the cloud right you're connected through something right and you know you may be running something for a very long time there's some you know people run stuff that's like that are servers right that are out there not just to run in 10 seconds but they're they run for days or something and then you need some sort of periodic view of what's going on you need to collect a profile so we have a way of telling the thing to emit periodic profiles and you can also specifically say hey send this python process a a Das Das off message which means turn off profiling okay so you can turn it on and you can turn it off and whenever you turn it off it now actually saves the profile so you can just always get a fresh profile just by doing dash dash off and dash dash on okay so it's like a toggle to you know have it output the performance and would it then give you the HTML file to yeah you can have it right um you know normally it produces uh an HTML file but it also can produce a Json file okay to make it easy to consume and it does it also produces one that's not interesting in this use case a kind of text based thing using the rich Library oh okay cool one of the things you mentioned there of somebody experimenting on this you know potential program they want to profile and they're running it in Jupiter and I think it was the Pyon 2021 talk said well we're working on the Jupiter thing it's not quite optimized and working well now um H how's that progressed and how's it performing now yeah so I mean it I don't recall exactly what I said okay it was in the before times yeah sure but yeah so I mean it does work for Jupiter it the problem with Jupiter is getting in edgewise to be able to profile the memory consumption is it's we don't yet know how to do it it's just an engineering question but have not figured out how to make it work um or make it work consistently there's still a problem with Windows can't easily do the memory profiling on Windows it's just that Windows makes things really painful when you want to do this kind of instrumentation it's super easy to do in Linux and the mac and it's virtually impossible to do on Windows an impossible really hard but uh but yeah the Jupiter thing we can't get the memory profiles out yet from Jupiter which is annoying okay I'd like to get that resolved but but it works fine at the command line okay yeah it sounds like any kind of project like this you now have a bunch of unique potentially software or operating system or whatever specific issues you got to kind of delve into so I find that very interesting I think like one of them that you hit along the way was trying to get it to behave well with K as far as like installing it and so forth and has that Journey gotten further along too oh yeah K's we have cond it works that uh as far as I'm aware everything is good on that front I mean the the you know the the python ecosystem is just it touches everything and it's it's huge and so you know we've had to learn a lot in this process like you know you're like oh blah blah blah pip fine that's not so hard and then oh now we're you know we need to actually transition to Pi Project because we're using setup and then there's K and we didn't know anything about cond and how cond works and you know those those folks are really busy so getting them to to help out it was a lot of waiting I mean I'm very graceful for their help but uh but yeah it's you know for for me and my students it's been it's been a great experience I mean I have to say the python Community is awesome in general super friendly super helpful uh super open it's been a joy to see scalen get adopted uh and you know get so many users but yeah you know I mean every now and then we will get some requests like you know this doesn't run on uh you know uh potato Linux 4.7 on you know the Raspberry Pi from the 80s and right you're like I'm sorry I can't help you like right there's so many edge cases there yeah yeah so we're we're trying to hit the big ones I mean I would love to get Windows uh support better Windows support but it's just a a technical limitation so going back to the idea of sort of results in figuring out areas that you could improve your program what have been some of the lwh hanging fruit of performance that you found along the way people have reported to you after using scaling so I mean I think the big ones have been really this gosh there are a bunch of them but so we have a bunch of case studies that are on the website and they do vary okay a lot of it comes down to inefficient use of the libraries which is I guess obvious in retrospect but it wasn't what we anticipated when we created uh scaling so you know if you use numpy nump can be screamingly fast but if you're doing something where you're like oh I'm using a for Loop and I'm doing numpy stuff well then you're moving between these worlds you're moving between the super fast C C++ world and python you're also not giving numpy the opportunity to like do a bunch of work on a giant object which can do very very fast while it's in Sealand it can take advantage of um a vectorization which is you know these right super super fast you can get giant speed UPS by doing Vector processing on your chip as opposed to doing things one at a time and so we see a lot of that okay where people are just like oh I was doing something and it turned out it was inefficient spending a lot of time in native code and had poor utilization and it's like well that's that's what I need to work on one of the most recent additions is is to add sort of hooks into having AI help potentially give advice how's that process gone for you and um what are the types of device it it it can give to a potential you know person who's looking at poor performance from their program yeah so it's been uh so it's been super exciting I mean I cannot stress enough how revolutionary this technology is by which I mean these large language models like GPT 3.5 and four the so we've uh Incorporated this into the user interface there is some work that is not yet incorporated into the user interface that one of my PhD students uh Sam Stern is working on which will push this even further okay but the user interface already uh allows you to basically select either a region of code or line of code and uh and say hey make this faster give me a suggestion to optimize this code and you have to provide your own open AI API key but it's not that expensive you know I always tell people it's like it's worth a quarter you know click a button uh the Magic Machine figures out how to speed up your code this is definitely a good use of your time and yeah you you know many many times it will come up with proposed optimizations that are correct optimizations they preserve the semantics of your code but speed things up by you know 10x or 20x or 100x and you know I you know sort of know my way around numpy and pandas and you know I use them but you know it will generate incredible code that just speeds things up tremendously and you know it's really just the click of a button so all things that we described in our paper and all the case studies that we had the code for we tried with the uh the AI and it was able to find as faster faster optimizations automatically so the paper you're referencing and I'll include a link for it is the triangulating python performance issues with scaling is that the one you're speaking of that's the one exactly yeah cool yeah I'll definitely include that and all the other talks we kind of mentioned this is a question I brought up when I had Pablo gindo sagato on when we talked about memory and as again the audience I have uh varies uh depending on the topic but it's usually kind of intermediate people that are wanting to learn more and trying to figure out how they can use these tools and one of the things I thought about was let's say you do get that new job at at a company and you have this task of like learning a code base and learning how to work in it and so forth do you think a profiler would be helpful in that process and what kinds of things would it help you discover yeah that's an interesting question I mean you know profilers are a kind of program understanding tool but they're not for understanding you know what the code does functionally it's really about understanding what it does um in terms of its performance right uh you know where it's let's let's say uh it gives you a resource oriented view of your code like where it's using the CPU and where it's using the memory and so on you know it's not obvious that this is always helpful to understanding a codebase but you know if you have code and spending a lot of time in some chunk of code certainly that code matters right this is the key code that's doing all the work yeah and so I would I would think that it would be helpful for understanding those things even if it turns out that well that that work this is as optimized as we can possibly make it uh you know I can't speed it up fine but if your goal is just to understand well here's this program where is it actually doing work that seems like it would be useful you know you probably don't care that much about what it's doing at startup or at the end of execution uh you probably don't care that much about you know rare cases uh at least if you're just starting to Grapple with a code base so it's an interesting perspective I really like that was not our Target for this system and but uh but it is it's interesting to think about it's it's provocative yeah I was thinking about the technical paper that you wrote there's like a and we kind of mentioned this at the beginning of it this this there's like a subset of conferences that are more specific to computer science conferences and the technical paper describing this won the best paper award at osdi which I'm not familiar with what's your experience of of uh these computer science conferences and the types of topics that they get into versus like say people that be more familiar with like Pyon or other python specific conferences yeah so so osdi is one of the two big quote unquote systems conferences in in academic uh computing so you know if you're a computer scientist and you have new results that you want to publish and uh you know you think they're important you try to get them published in these you know these top most selective venues so you know you've probably heard of like nature magazine or Science magazine or you know New England Journal of Medicine this is a similar sort of thing like you know these are high highly prestigious highly selective venues uh for publishing research so you know the difference between publishing research and and and public and you know giving a talk in an industry conference that you know industry conferences are not really necessarily about something that's new like a new idea or a technical Advance like if you have something that just has a better user interface or it's just better in some way provide some functionality even if functionality is well known in the scientific literature having a useful tool or telling people about the tool or showing them how to use it and all of these things are valuable right so you know I I'll try to give you an example I don't know if you've heard of map ruce sure from back in the day at Google yeah so you know the map ruce paper appeared at osdi which is the same conference either osdi or sosp there's these two uh systems conferences the paper describing tensor flow appeared osdi right so there are you know these conferences often have papers that are about big systems often