Swifter Speeds Up Pandas DataFrames

Key Takeaways

what is going on guys welcome back in today's video we're going to learn how to speed up panda's data frame operations using swifter in python so let's get right into it [Music] all right so swifter is a python module that allows us to use the pandas apply function in the fastest possible manner it is based on another python module called desk and depending on the task it determines if the ordinary apply function of pandas is faster or if a multi-processed multi-threaded way of solving the problem is faster and then it chooses the respective way to speed up the operation that we're trying to do now let us get started by installing the libraries that we're going to need today first of all we're going to need pandas itself so pip install pandas if you don't have it yet then we're also going to use in this video the pandas data reader however you don't have to use it if you want to use a different data set i'm just going to work with some stock data here you can use a csv file that you just read in with pandas you can create your own data whatever you want to do i'm just going to use the pandas data reader to get some stock data so we're going to say pan pip install pandas dash data reader and then finally obviously the topic of today's video pip install swifter there you go you can see it has quite a lot of dependencies once we have that we're going to work today in a jupiter lab environment here again as always you can use whatever editor you want uh we're going to start by importing here pandas spd let me maybe just zoom in a little bit uh and then we're going to say that um maybe we should do some more imports here pandas data reader is going to be imported as web and then we're going to also import date time as dt which is a core python module and we're also going to import time which is another core python module um and then what we're going to do is we're going to load some stock data we're going to say start equals dt date time dot or actually not dots 20. actually let's do 2000 first of january and the end is dt daytime dot now so the current date and time stamp and then we're going to say that the data frame is web dot data reader uh let's pick a company that exists for longer so let's go with goldman sachs for example yahoo is the finance api and then start end is the time frame that we're looking at uh dt is not defined there you go now it is defined and if we look at a data frame this is what we have again you can take whatever data you want i just want to have some some numbers here we're not going to do anything useful here we're not going to do any stock application here i'm just going to take the numbers and do something that takes time like taking the factorial off some of these numbers or even some of these numbers multiplied by another number just so that uh the function that we're going to write wastes some time so that we can see the effect of swifter but let's start with the most basic example of wasting time let's just sleep let's just say time dot sleep we're gonna say we have a slow function here uh we have a row that we get from pandas or from the data frame we're gonna say time sleep 0.001 seconds uh and we're going to say then um return the row as a result so that is our function and uh what we're now going to do is we're gonna measure the time off in operation so the operation is we take from the data frame the closing price and we want to see the first thousand closing prices so this is the data frame we just have uh here a couple of uh of closing prices and we want to apply now to them a function so the apply function basically takes each row and applies a function to it so when we pass here the slow function uh what happens is that each row will be passed here as row and then we're gonna in this case just return the row nothing too fancy we're gonna get the exact same thing out here but of course we can also say row squared row something else we're just going to return the row in this case and we're going to measure how long it takes we're going to say t1 equals time time we're going to copy this we're going to say this is t2 and then this should take in my prepare code around 14.8 seconds so this will run now you can see uh essentially what it does is it executes this thing a thousand time it sleeps for for 0.001 seconds a thousand times and returns zero now you can see it's done we can say t2 minus t1 15.65 seconds uh was was the time it took and now we can see if we can speed that process up with swifter and since swifter does multi-processing slash multi-threading obviously this is going to be faster especially because sleep is a function that can be easily used for so so the downtime can easily be used it's not even a computation it's just waiting and in that time we can do something else so let's say swifter import swifter we're going to say here again t1 equals time time we're going to say t2 equals time time as well and then we're going to say here the same thing we're going to copy that line the only difference here is going to be that after the i lock we're going to say dot swifter this is how we apply the swifter function we or the swifter module we take the same thing we take the data frame piece that we want to look at we say dot swifter and now everything we do is going to be um optimized by swifter so if i run this now uh this is already done i can say t2 minus t1 you can see 0.649 seconds um compared to 15.65 seconds so obviously swifter massively speeded up this uh or sped up this what is what is actually the past tense i'm not sure um but this was um way faster than before and you can see that swifter actually worked now let's look at an example that maybe involves some computation because it's easy to optimize time dot sleep so let's go ahead and say we want to have um a lambda expression here as a function so for this we're going to import from decimal which is also called python module we're going to import decimal with a capital d and we're going to say again t1 equals time time t2 equals time time in between what we're going to do is we're going to say df close apply and we're going to apply simple lambda function we're going to take the row x and we're going to say i want to have a decimal value of math we need to import math as well also part of the core python stack math dot factorial of the integer value of this row times 50. so essentially we take the closing price doesn't really make sense what we're doing here i'm just doing something that takes time um but essentially we take the value we multiply it by 50 we truncate the decimal places we get the factorial of that large number and um yeah we turn it into a decimal and then we return it basically that's all we do and this is going to take quite some time so i'm going to run this and probably we have to speed this up because this will take around 67 seconds here all right so it's now done we can see how much time it took t2 minus t1 67 seconds now let's go ahead and try the same thing that we did here with swifter so essentially just the same thing but dot swifter here dot apply this will also not be happening in a second but uh in the prepared code it took only 37 seconds compared to 67 seconds which is way better there you go it's now done you can also see that we have this progress bar obviously when we do something with multi-processing here uh it's going to to have some overhead that uh will take some time but sometimes it pays off in this case t2 minus t1 we can see 34 compared to 67 seconds definitely a speed up so here you saw that swifter chose this operation here uh or chose to use das in this case so let's look at one last example not really a fancy example just an example that is uh way faster just squaring the numbers to see that we also see a speed up there time time and t2 as well then we're going to say here df close apply lambda x x squared so we're just going to return the squared values this took t2 minus t1 district 0.002 seconds whereas if i take this same thing here and i say swifter okay this actually yeah a little bit faster but usually the thing that happens is if swifter notices and i have a a nice graphic here do i have it here there you go um this is from the github page what happens when pandas uh when swift or sorry recognizes that ordinary panda supply is faster than the desk then it's going to just pick the pandas right so it always finds the fastest possible way so you can see here we have certain graphs that say okay the lower um the line is or the lower the point is the faster so the execution time is the y-axis as far as i remember so in this case panda supply is faster desk applies pretty slow and the swifter applies also slower than the panda supply but as we go on with the size of the operation for example you can see that the panda supplies actually the the slowest version here whereas the swifter and desk are faster so this is definitely something that is happening behind the scenes we submit an operation we want to do some apply operation and swifter figures out if the ordinary pandas apply function is enough or if it has to use das apply or something else so that's it for today's video i hope you enjoyed it and hope you learned something if so let me know by hitting a like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you much for watching see you next video and bye [Music] you