150x Pandas Speed-Up: cuDF in Python

Key Takeaways

what is going on guys welcome back in this video today we're going to learn how to massively speed up working with panda data frames by using a GPU data frame Library called CDF which is designed for fast and efficient data manipulation on gpus from Nvidia so let us get right into [Music] ited all right so we're going to learn how to massively speed up working with Panda's data frames in this video today by using a Library called KF which is designed for NVIDIA gpus and allows us to use our gpus to do data manipulation leading to a massive speed up when working with data frames now I do have some code here prepared to show you how this works and how much faster this is but first of all we need to make sure that we have the proper version of the CF package installed and for this we need to know what Cuda version is running on our system now of course this only works if you have a GPU that supports Cuda and if you have such a GPU on Linux you can run Nvidia DM I just Google how it works on Windows if it doesn't work in the same way uh and in my case here it says Cuda version 12.5 so I know I need the package for Cuda 12 so what I do is I type pip 3 install cf-c2 now if you have Cuda 11 you install cf- cu11 and so on so I already have this installed on my system and once you have this installed basically all you have to do is you have to replace pandas data frames by qf data frames nothing else changes and you have a massive massive speed up um in your operation so this first file that I have here what I do here is I Generate random values I have 10 to the^ of 8 rows which is quite a bit and what I do is I Generate random values I also have IDs and then I take this dictionary that is the result of doing that and I turn it into a pandis data frame and into a CDF data frame and then basically I just Group by ID and aggregate the value column so I just sum up all the values I do that here for pandas on the CPU and I do that with KF on the GPU and KF also uses the CPU if the GPU is not available or problematic or not reasonable or something like that um but I also have to append here sort index because since the GPU is doing everything in parallel the order is messed up a little bit but this is not really a problem so I just sort it and that's the only difference and for these two functions I measured the time and when I run this now you will see that for the pandas data frame uh it takes 1.1 seconds for the qdf data frame or for the coup data frame it takes uh 0.07 seconds which is a speed up of 15 or in percent of 1, 143% this is massive um this is an extreme speed up and I think I saw some where uh the claim that you can get a speed up of up to 150 so 150 times faster using kdf now I have a second script here main 2py uh where I load an actual data set uh which is the yellow trip data so it's a taxi data set uh in park format uh I load it here in pandas I load it here in KF and what I do here is some basic stuff like dropping all the Nan values um filtering for a trip distance larger than two and then also again grouping and then uh applying the mean operation or aggregation and the sum aggregation for these two columns and then I sort the values here by trip distance and then I get the result I do the exact same thing here for the Cuda data frame and if I run this code you are going to see after some time that the panda time was 0.46 seconds the CF time is 0.1 seconds so here we have a speed up of 4.5 so 4.5 times faster and in percent this is 355 per. so you can see that these operations are much faster using qdf even though nothing else changes so I just take the data frame and I turn it into a KF data frame or into a Cuda data frame you could say uh and it's much faster because it utilizes the GPU now one thing that you have to consider here is when you want to compare those values the resulting values for a equality or in terms of equality you have to do it with NP all close because these values are basically the same but not exactly because there are some slight differences here when it comes to floating Point numbers we're talking about very very small differences I actually print them here in the end um we're talking about something like e -10 so something very very small but we have some small imprecisions when it comes to floating Point numbers because the arithmetic the floating Point arithmetic is done differently on a CPU and on a GPU so we will have some small differences which means that if we compare for exact equality we're not going to get a true we're going to get a false but we have this function NP all close to basically uh see if the values are close enough and in this case there are I do the same thing here for the individual value strip distance and total amount so we can see we get the same results uh with some slight very very small uh differences but it's not necessarily because the operations are wrong but just because these small inconsistencies or the small imprecisions on a floating point or in the floating Point arithmetics are just different ones they're also there with a CPU but they're just different ones on a GPU so this is how you do that but there's an even simpler way to do that in Jupiter notebook so in Jupiter notebooks all you have to do is you have to load an extension and this extension uh will allow you to not even have to use KF at all so if I open up actually let me navigate here to the current directory um I can open up a Jupiter lap instance here and um I can copy paste some code so I can actually go and uh let's actually use this code here let's copy paste that go to the browser open a new jupyter notebook save it as main ipynb copy paste this and now let's say I don't even want to have these different um these different uh data frame types I can actually remove everything that has to do with CF I have my basic pandas code so I can remove also this here and I can remove also this here and I can remove this here and also the speed up calculations and all the comparisons so I just have the operation in pandas I can run this and we will get something like 1 point something yeah 1.09 seconds now the great thing that I can do here is I can add one line to my Jupiter notebook which is percent load extension and then just KF pandas and by doing that it automatically uses qdf and not pandas anymore so you can see that now the operation took 0.07 seconds even though I don't use CF at all I don't use any CF explicitly I just have my pandas code without even changing anything about it and just by saying I want to use CF p pandas it is using KF pandas and I have a speed up which is massive uh now the best way to to just uh you know disable this again is to restart the kernel so that's the best thing that you can do if you want to or actually I don't even know if that's going to solve it maybe I have to even start the Jupiter laap instance no actually restarting the kernel is fine uh but yeah if you say load extension in a Jupiter notebook KF panda you're automatically telling your environment or your Colonel to use KF instead of pandas and to use the GPU when possible which will leave to a which will lead to a massive speed up so that's it for today's video I hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this Channel and hit the notification Bell to not miss a single future video for free other than that thank you much for watching see you in the next video and bye for