How to make Pandas extremely fast without changing any code.

Key Takeaways

let me show you how to make pandas many many times faster without changing your code this is probably going to be one of the best things that you learn today like if you are in the python World obviously you know pandas is probably the most popular library to manipulate to process data so I have here a quick example where I'm just going to load the data set is a large data set and we're going to run some queries and we are going to measure the performance of pandas on this data set so I'm going to start here by just importing pandas and as you can see here I'm printing out the name of the library that I'm currently using so that's pandas this is going to download the data set and the data set is the New York City parking data set from 2022 is parking violations it's a pretty big data set so there like I think it's like 1.5 million rows that we have in here so this section is just going to read this parquette file this is the name of the file and we're only interested in loading these five columns here in memory and then I'm going to print the first few rows from that data set and notice how I'm timing every single operation here like as we go forward I'm going to keep grabbing and recording how long it takes every one of these operations I'm going to run this it's going to take a couple seconds so here you get the first five rows uh printed out and I stored how long it took to do that operation in this variable here that we will later use okay so let's do now let's run a few queries this first query here will return which park in violation is the most commonly committed by vehicles from various US states so let me run this and we get all right so here are the five first elements here so you get the violation the description Etc uh the second query uh and by the way again I just timed that query and I stored it in query one then the second query will return which vehicle body types are the most frequently involved in parking violations so let's run that turns out that Suburbans are the first one followed by four doors SED Dan so who would have thought and the final query is how do parking violations vary across days of the weeks this is probably the most complex query the query that's going to take the longest here's the query here I'm printing the again the five the top five rows and then I'm timing that query uh so here we get Sundays and Saturdays and Mondays Etc so the final sale of this notebook here just takes all of these values the time it took to run every single operation and it sums them so we get how many seconds total uh it took pandas to do this and again this is just a very simple example you can think of like if you have an application that's using pandas in a more involved way with more data obviously you you can take an idea of how long that's going to take so the total time here was 12.54 seconds so I'm going to write down down that down here just so we remember that was 12.54 seconds and now comes the trick okay so I'm going to be using fire dots so fire dots is here in GitHub I'm going to leave a link to the repo somewhere here far dock is going to accelerate our code without having to change anything that's the important thing so I'm going to add here a new Cale put it at the top and let's install baru so that's going to be quiet uninstall if it already exists baru sorry I made a mistake here so I'm going to install the paru library and then I can replace this line so instead of importing pandas I'm going to import from fir do. pandas aspd and I'm just going to commend out this one here I'm going to run now so now every time I use PD I'm actually using the panda implementation of fire dots which they share the same interface so nothing in my code has to change so let's rerun everything here just going to go through all of it so you see what the performance uh optimization was by run this that took 2.81 seconds so 2.81 if I make that division that's 4.5 times faster without changing anything else literally a single line I was able to optimize my code 4.5 times make it 4. times five uh five faster this is a no-brainer if you have an application if you're using pandas check out firo again this is a there is a a GitHub repo here explains everything that you need to know they have more examples here they have compatibility charts Etc uh it's not only faster than pandas but it's even faster than polers which is an alternative to pandas and the good news with fire dos is that you don't have to change anything their website says that it could be up to 48 times faster than pandas I think this is an no-brainer I hope you enjoy it and I'll see you in the next one bye-bye