Pandas 2.0 : Everything You Need to Know

Key Takeaways

the recent stack Overflow annual survey shows that over 25 percent of all respondents have some experience with the python package pandas and that's more than 50 percent of all python users with the recent popularity of Alternatives like polars some have been pondering if pandas will be around for the Long Haul but in this video we're going to talk about the brand new pandas 2.0 pandas is so widely used that most of the updates to the package are done in the background and you don't really notice them when they roll out and actually it's pretty much the same for pandas 2.0 but there is one big change that I foresee being highly adopted in future versions of pandas and it's the new Apache Arrow back end in a recent article by one of the core developers he goes into detail about why this change is such a big deal so in this video we're going to talk about how this integration into pandas 2.0 will make the library simpler faster work better with related libraries and overall help the library move into the future okay so I'm here in a Jupiter lab environment if you want to learn more about Jupiter I have a whole video on it at the time of filming this video pandas 2.0 is still in pre-release but you can install the pre-release version using this pip install command and be sure to check out the release documents for pandas 2.0 for full details on everything that's changed in this version let's load the important libraries import pandas I've also installed polars and let's print out the version so you know which ones I'm using and there you can see I've installed the pandas 2.0 release candidate zero so let's read in a CSV file as a data frame when you're reading in this file now pandas is storing that data in your computer's memory and historically pandas has used numpy as the back end for storing most of its data types so I'll run a head command so you can see the data in this data frame and if I run df.info you can see that each of these columns is stored as its own unique data type the numeric values are stored as in 64 and the string values are stored as objects now if we focus in on one of these columns that has numeric values we can actually do a DOT values on just that column if I run a type on this column you can see that it is indeed a numpy array similarly if I just create a panda series from some data I see that it's automatically converted into a in 64 or if I create a series with some strings it will store these as an object data type so by default nothing's really changed but with pandas 2.0 we have some more options by explicitly stating that we want pandas to use the pi Arrow data type we can see that the series has a N64 with the pi Arrow back end similarly this series of strings is now a data type string instead of a general python object if we want to we can tell pandas to use Arrow by default and we do this by setting the option mode d-type back end as Pi Arrow now let's go back and read that same CSV as before using the pi Arrow back end so I'm going to call this DF Arrow we're also going to set nullable d-types as true now when I run dot info on this data frame we can see the data types are using pi Arrow by default when it reads in the CSV so I know what you're thinking okay Rob you can use Pi Arrow back end but why use Arrow let's talk about some of the advantages one reason is its ability to handle missing values remember this Panda series from before where we have a bunch of integers what if one of these values happens to be a none type now historically these data types would then automatically be converted into float types I'm not going to go into all the detail as to why but just know that this is not ideal in all cases but let's see what happens if we use the pi Arrow d-type now we can keep the data type as a int 64 and it can handle the fact that there are null values this might not seem like a big deal in itself but it's important to remember that this is actually using Apache arrows implementation of dealing with missing values so the fact that pianos can use this natively can make things a lot more efficient the next reason why we would want to use Pi Aero back end is because because of speed now this will depend on what machine that you're running it on but let's just do a few different tests to see the speed difference on my machine between a few operations on the data sets we have so we can take this count column and run a mean on it and using time it we could see how long this takes on average and let's compare that to the pi Arrow back end so with a simple operation on a small data set we're already seeing a slight Improvement in speed let's compare the speed of just reading in the file itself Now using pi Arrow backend with the use nullable d-types and there we see a pretty significant speed up in the load time so ran this a few times but you could see that loading it with arrow takes about five percent of the time as it would with the old numpy backend and the same goes for string operations because Pi Arrow has its specific string data type let's compare the time it takes to see which name in this data set start with the letter a and now with the pi Arrow back end so in this case it's about 30 times faster using pi arrow and the next reason is interoperability and that's just basically that arrow is used as a backend for more than just pandas R spark and polars all you can use Arrow back end in some way a good analogy to this is how CSV file formats when saved to your hard drive can be used by many applications pandas Excel are and that format is independent of the programs that use it now here we're talking about data that's stored in memory so it's a little bit different than the file extension of CSV but it's similar in the way that many other programs like R spark and polars use the Apache Arrow back end and this makes transferring of data in memory between those programs a lot more efficient so let's Show an example remember how I imported polars before I know that pullers is pretty good for building Big Data pipelines but let's say I've read in some data into memory using pandas for instance using read SQL I have this data frame and I want to run some pullers operations on it but then I want to convert it back into pandas so that I can plot the data the nice thing about using the arrow back end is we can transfer that data in memory between the two packages and do it efficiently we've loaded in this Arrow pandas data frame and we can use pullers from pandas to read this in as a polars data frame now we can do the data intensive work like some group by aggregations using polars and we could take this aggregated data frame and convert it back into pandas so that we can leverage pandas built-in plotting methods now in this example it might have been easier just to do everything in pandas but when you're working with large data sets the fact that you can move them back and forth with very little overhead is a huge deal and the last reason I want to cover about why the Aero back end is so important is Aero data types as Mark writes in his article about this Apache Arrow allows for some better data types than what you can find just using numpy we could show just a simple example of this the Boolean data type in numpy actually uses eight bits to store this Boolean value Pi Arrow however only uses a single bit which is much more efficient not only can we see here that this uses less memory but as we discussed before the pi Aero version allows for null values if we tried to do this with the numpy back end it would convert our data type to a default python object in this article Mark shows how you can set custom data types like this where you have a list with strings within it unfortunately these don't render well in a Jupiter notebook environment but I can also show you that this date type column is actually a date 32 and these have similar built-in methods to what we normally would use with date time or we can do things like day of year or day of week and it's with custom data types like this that are built into arrow that I think pandas 2.0 is really going to push us towards using more and it will make our data processing much more efficient and standardized thanks for watching this video on Panos 2.0 like subscribe hit the Bell button follow me on all the different socials you can check out a link to everything that I discussed here in the description and like always give me your feedback in the comments below see you next time