Python for Data Science #4: Mastering NumPy and Pandas for Data Manipulation

Key Takeaways

Hello everyone and welcome to another week of Python. Since you all know that we are learning how to grasp a good understanding of the world of Python so that when you will dive you know in the world of LLM geni you become comfortable from the coding point of today in our week four of Python development we're going to be covering some amazing topics that will become kind of a you know day-to-day activity for all data scientist working with packages like numpy pandas and you know doing all these uh data activities of slicing data uh you know filtering data merging data uh cleaning data is going to be your day-to-day task. So in today's session we're going to be learning about the wonderful power of numpy and pandas. So for that uh you will get all these resources. Don't worry about that just enjoy the course. So what I'm doing is as you know that uh we in the world of Python especially working in the notebooks we import certain packages that makes our life easy and in that journey we are learning about numpy and pandas today. Numpy is a package that allows us to do a lot of mathematical and statistical work. And on the other hand, pandas allows us to do more of a structural data frame related works where you upload data frames like CSV, Excel, you know, tab eliminated, comma, separated files and play with those datas, either cleaning those data, observing those data, visualizing those data so that once your data is ready, you can feed that to any machine learning model or maybe you know process for further data. pipeline. So we'll start with numpy and then gradually we'll go towards the pandas world of it. So most of the time when you in you know download Anaconda or you're working on Google Collab or you're working on ids like Visual Studio and you install Python uh numbers and Vas get automatically installed in the case of Anaconda and Google Collap. But if you have downloaded Python uh separately then in those cases you need to import these package. So all we have to do is do pip install numpy and the package will be installed. Once it is installed we import the package import numpy as np. So this np that you see is an alias in python. We have some standards. I mean here I can give literally anything. I can type j also but there are some you know industry practice and we are following those. So np is the industrial way but good coding practice to give an alias to numpy. So how does it work? As you can see on my screen I'm defining a array numpy that is np dot array and it declares a one-dimensional array for me and saves it in a variable called ar r. Then if I pass two square bracket elements you know you see first list and second list separated by a comma then it becomes a matrix. How does it look? Let me run this cell in a different set. Right? So if I run this thing so you see this is my one-dimensional array and if I print my two-dimensional array. So you see so you got a like uh you know two rows and three columns data a matrix this allows us to play with different dimension of data we can do a lot of wonderful things you know like I I can uh let's say if I have arr I want to know what is there at the zerooth location right at zero location or the index value of zero I have one see this one I got printed If you want a range of data like from certain elements to another element. So starting from position one this is in Python the coding starts from the counting starts from zero. So 0 uh 1 this is a position. Okay this is two this is three and this is four. So what I'm doing is from one position till n minus one. So 1 2 3 three elements got printed. Similarly, I can fetch data for a matrix also instead of a single array. But in matrix, you will get the entire row because it's a two-dimensional array. And in matrix also you can fetch the uh range of values too. Right. So at this location at this point this number got saved. Got it? Uh we can do some conditional checks also. What does conditional checks mean? For example, if you look at here, I want to check even numbers whether they are even or not. So in the entire array, I have 1 2 3 4 5. So what I'm doing here is element divide by two the remainder should be equal to zero. If I divide 1 by two remainder will not be zero hence that did not get printed. But 2 divided by two remainder zero hence two got printed. So these have some kind of an operations you can do with the package as number. Then there are certain advanced thing that you can do which is called your vectorzation. Converting your data into vectors and broadcasting. Basically broadcasting mean when you want to make a change to an entire you know numpy array or a matrix. So let's define these two metrics x and y. Uh what if I have to add them right? So what will happen? One will added to 10. Two will get added to 20 and three will get added to 30. So you see x + y will give me the sum. Simply multiplication also 10 into 1, 20 into 2 and 30 into 3. You can broadcast also. What if I want to uh square each and every number 1 square, 2 square and 3 square. Uh we can arrange and reshape also. What does arrange and reshape means? What happens is? So you see I have an array you know I'll declare an array of size six right and then I'm saying that convert that into a matrix of two rows and three columns so you see this thing m becomes an array np range six values if this is confusing you can see this like this I'll add a cell here and run it. So nparrange what it has done it arrange it has started counting from zero and given me values till five. So I declared an np array first and then reshaping it to 2a 3. Okay. So this way I got this matrix and similarly another vector. Then I can add these values. So what has happened? 10 0 gives me 10. 10 + 3 gives me 13. 20 + 1 21. 20 + 4 gives me 24. Right? Or I can do sum also. M dot sum. What it will do? It will give me the sum of all the numbers. 5 + 4 9 9 + 3 12 12 + 3 15. So you see 15 here and at the same time you can even calculate the mean and median and some mathematical uh statistical information also that could be really helpful right and similarly just like in numpy help us with mathematical related operations pandas help us with data at a table form you see it's called panda series so import pandas as pd E then I'm defining a series these are the values but in bandas I have to give the index also right so if I run this cell you see a panda series got defined with these values and these index if I do type see it's a panda series and what is happening in this cell I'm defining a series and then printing it. So you see this the series got printed. Then just like your numpy we can access element here also we can call element by their index. So if I do B sB B 20 will get printed or you can print multiple values also but this is not an ideal use case here I would have preferred numpy then why do we need pandas when you have complex architecture like this you have a dictionary keys and values key value key value so when I pass a dictionary to a data frame so this is how it gets converted You see that this becomes my first uh column. This becomes my second column and this becomes my third column. And these three things becomes my column names. So column name name all the value. Column name age all the values and column name score all the values. You can fetch one entire column by writing df.h age. If I write it here, see entire column got fetched. You can fetch one element also or one entire row. So when I did this iOS it gives me the location. I log zero means in this data frame at zero index this value is there correct? At zero index. So I log gives me the index value and then you can apply a filter too. Filter means I want only information where age is greater than 30. So when I run this cell see only those rows got printed where the age is above 30. But this is good when you only have one table of data. What if if you have multiple data frames, multiple data sources, then there is a chance you have to join them, merge them and that is what you'll do now. So let's create another data frame, data frame sales. So now I already have one data frame here, data frame df and I created another data frame called sales. Now I'll merge them. That means data frame join outer join just like SQL. Okay. On condition is name. Wherever the name will match this will be joined. So if you see the data frame merged this is how it looks. This merge data. Now you want to add one more column is senior. That means whenever someone is above 30 years I want to give them a tag of senior. So what I did data frame column name and added the condition. Then what I want to do is grouping by on the basis of whether you are a senior employee or not. Data frame group by column senior. I want to see the average salary. Okay. So people who are you know less than 30 years of old their average salary is 87.5 and people who are above 30 years old their average is also 37 87.5 because this is a mockup data right but you got the idea grouping by on a condition right so not only you can calculate mean you can calculate other things also. So you see people who are u not senior their maximum salary is 90 and people who are senior their maximum salary is 95. However you might have noticed something. What is this? This is called nan null values. When computer or compiler does not find a value it assigns a null nan value directly. In Python we can handle them. For example, I'm declaring a new data frame called DF2 with intentionally adding some non values. Non values. How do we check null values? Data frame is na means data frame and a function that checks meth or or or a method that checks whether a particular value is none or not. If it is none it increase the counter like it the counter becomes one and I'm doing a sum of all those ones right so see this thing there is one column one value within under the column name with none there is one row with age as none and there is one element in score with none got it then there are multiple ways to fix this thing. What if I want to fill the missing values with main value? What will I do? I will say that in the original data frame in the age column, wherever you see null value, replace that with the mean value. So if I show you the mean value is 28, right? So this is where the null value is there. This is what we want to uh you know trying to replace that with the null value with the mean value basically right or what if you want to drop them like you know I'm not interested just simply drop the null values see it has dropped the uh null values and giving you the output right and what if there are duplicates you can drop them also by df dot duplicates and it will drop the duplicates values right so you see null values can got replaced with 28 or I can delete the values wherever none is there do got deleted This got deleted. This got deleted. Right. And this got uh this got deleted. Or if there are duplicates, I can drop also. If there are duplicate patients, students, employees, I can delete them also. I So these are some amazing tasks that you can do when you are working with numpies and partners. So if you found this interesting and amazing, trust me this basic fundamentals that we have done today will be really helpful for you going forward. So stay tuned. We are going to cover amazing chapter next week that will cover your visualization. Until then I'm Jan Maharaj. Thank you very much.