Python for Data Science Full Course 2026 [Free] | Learn Data Science With Python | Simplilearn

Key Takeaways

Hey there, welcome to this apply data science with Python course by simply learn. If you've ever wanted to dive into the world of data science and learn how to turn raw data into powerful insights, then you are at the right place. Whether you're a beginner or have some experience with Python, this course will take you key real world data problems and by the end you will be comfortable using Python libraries like NumPy, Pandas and Matt plot lib and you will have a solid understanding of the data science pipeline. So here's a sneak peek of what we're going to cover in this video. First we'll start with the basics of Python libraries including numpy, pandas and matro lib which are the backbone of any data science project. We'll also walk you through the data science pipeline a road map for solving data problems. Next we will dive deeper into pandas learning about series data frames and lambda functions which are essential for working with and manipulating data. Then we'll move on to some advanced pandas techniques including data inspection, column operations and how to aggregate data. You'll also learn how to visualize data with map plot lib and explore different chart types. After that, we'll jump into exploratory data analysis EDA where we will explore charts, histograms, scatter plots, and even heat maps to uncover pattern and insights. We'll also cover key statistical concepts like central tendency and inspiration and tackle data cleaning and outlier management to prepare you for data analysis. Finally, we'll explore more advanced concept like skewess ktosis and data prep-processing. We'll also touch on probability which is the foundation of many data science models. Also, if you're looking to kickstart your career in data science, then I highly recommend checking out the data scientist masters program in collaboration with Microsoft and Semilot. This 11mon program offers live interactive classes led by top industry experts giving you hands-on experience with AI powered tools like Python, SQL and machine learning. You'll also work on real world projects including a capstone project earning Microsoft certificate along with simply learn data science certificate to showcase your skills. So whether you're a beginner or looking to upgrade your skills, this course is perfect for AI enthusiasts, students and recent graduates. And the best part is you will get career support to help land your dream job. Enroll today and start your journey towards becoming a certified data scientist with real world expertise. To get started, here's a quick quiz question for you. The question is, which Python library is best for data manipulation? Your options are NumPy, Pandas, Mattplot Lib, or Cort. Let me know your answers in the comment section below. So without any further ado, let's get started. We are going to get started with this now. See, let's understand the journey to data science. Right? So, whenever you talk about any data science project, there is a uh crisp DM framework that we follow. We'll be studying that later. But you are doing a data science project. So like this. So whenever you talk about any data science project, so it will be always going through these all things. I'll just let you know in short what I mean by that. So whenever you have a project from the client, the first phase is where you understand. So let's say your client is FISA. So you understand what are they into, what is their domain, uh what is their problem statement, what are the challenges that they are facing. So a business analyst comes over here and understands all these things. Then he propagates the requirement to the technical team who will be getting the data, right? Who will be getting the data and starting to analyze the data, right? Because in a data set, let's say FISA, right? So there will be all pharma related terms. As a data scientist you will have to understand each and every column and correlated with the domain knowledge because if you don't understand that you will not be able to solve their real life challenges. So data understanding comes then after the data understanding comes the data preparation. So data preparation is the phase where the data in real life will never be as clean as expected. You expect the data to be awesome but it's going to be worst. You really have to make your hands dirty for cleaning the data. So cleaning the data doing some visualization on the data. So this is where your libraries like numpy, pandas, mattplot, lib and seab bond comes into picture right so we are going to see this right so all of this comes into your picture and then comes your modeling this is where your machine learning comes into your picture. So the library that you learn is scikit learn and some other libraries but mainly scikit learn. If you are doing deep learning you learn tensorflow and then you get into evaluation of the model. So obviously first you analyze clean the data then you create multiple machine learning models because one model may not be the best. So you try to create many many many just like in a class as a teacher I'm teaching to all of you and then I am choosing the best student. Okay. So I have to send them to some Olympia competitions. I will choose the best student and that is what happens in the evaluation stage. So in evaluation ch stage as I told you you select the best model and after that best model is selected you get into the deployment stage and deployment means obviously you have to share it with the model so you productionize it. So today we are here in understanding these libraries and once you understand these libraries then you are good to get into this flow in the machine learning part not now but one thing I would like to definitely say that if a data science project you if you are talking about a data science project and if a data science project let's say is taking a total of 100 days to complete successfully then I want to say that 70 to 80 days go in phase 1 2 and 3 Then I would say 10 to 15 days go in 4 and 5 and another 10 to 15 days go in six. So although machine learning is so so important but for actually creating models it does not take that much time. The highest amount of time taken is over here for getting these all things aligned. Right? So this is something which is very important for anyone to realize. So data preparation stage is one of the most important thing. Okay. So that's how the overall data science pipeline is and also you know at times if you are not able to create a good model you can also see that you may have to go back. So let's say you do this thing in a little bit hurry and you try to create a model quickly. Your model will be worst worse something that you know client will just not accept. So you may have to go back and do this. In fact here also if you do any misunderstanding you may have to go back and understand the requirements. So it's not an easy job friends. So that's something important to be understood by all of us. Okay. Now we need not get into all of this. This was just to give you an idea. What is more important is getting started with the numpy library. But you know this helps you understand the overall flow. What are you learning and why are you learning? One of the model you know which is productionize uh looks something like this. So what are we essentially doing over here? I have hosted this model which is productionize using the streamllet app. So if you see this is a tick prediction app. We will be actually dealing with this data set when I explore for you the seaborn and mattplot li. So that data set we will be looking into at that point of time. Right now it's not important but this is that data set for a US restaurant where I am trying to predict what will be the tip that a customer is likely to give based on his or her total bill. How many people visited in all the gender of the customer who paid the bill? whether he smokes or not, the day on which he visited the restaurant and whether he visited for lunch or dinner. So you can actually play around with this. Let's say I go to the restaurant and my total bill is $15. Dollar say it's a US data set $15 and I went with my wife. So total two people I was the one who paid the bill. I do not smoke. I went on Saturday and I went for dinner. So if these are the parameters, I'm likely to give a tip of $2.45. And in the same way, you can play around. Let's say Dan went for a lunch time, then $2.55. If Dan smokes, if the customer smokes, rather than saying that, then this is the one. If I went ahead and let's say the you know this is not this is some female then the tip given is this and if it is a male then the tip given is this. So male gives relatively long a higher tip is what we understand and smoker is giving 2.36 non-smoker gives relatively higher tip. So these are the things that we understand. You can also take a moment and try playing around with this because this is deployed on the cloud. So one can try that out. Now this deployment part which I showed you know that comes at the end of machine learning or somewhere at the end of deep learning. Further more ahead how does the data look like? In this case it's a CSV file. It looks something like this. This is how the data looks like where this is the dependent variable which I'm trying to predict on the basis of other factors. Yes. So we'll be working on this data set as well later on. So now let's get started. Numpai, right? So, getting started with numpy. Um. Oh. So what is exactly numpy? Well, numpy is the backbone of machine learning and data science in Python. It is one of the most important libraries in Python for numerical computing. See, you need to understand num in numpy means numerical and pi means python, right? and numpy. Why numpy? I know lists. Why numpy? I know the topic list. Then why do I need a numpy array? Because numpy is super duper ultra fast. So numpy is almost 10 to 15 times faster than list. That is what you need in data science. Right? So numpy supports large multi-dimensional arrays along with the collection of the mathematical functions. Next, it is very powerful for doing this and it is super fast and efficient as it is implemented in C. Now C talking about Python. Yes. So if you you know just if I go back and if I say I want to write a function to check if a uh not check function to find factorial of a number that's how you write it and then while testing you test it like this right this is a recursive function to find factorial of a number that's how you write it in this. So when you are giving a call to this factorial num function then control goes over here and some logic is applied to do the things. Okay. So the function right if you see over here this function is in Python language. However, numpy is implemented in C andron and that is why it is the till today one of the fastest language is C only. C, C++ are the fastest language. So whenever I am going to call some numpy function then the body of that function is written in C. Mixed language programming it is called as cython. I'll show you uh can you show the body of any numpy function as it is written in C language. You can see over here. I think I did not give it properly. So, see whenever you are giving a call right. So how does Python use the and how is it implemented in the back end because whenever I'm saying sum it will find out the sum of the array. So in numpy finding the sum is this as simple as that. But in the back end the code looks like this. And in written when sum is called there is a python code which is getting executed right. So that is I want to say right. So that is what is called as Syon code. Python and C together you call it as Syon and it looks something like this. We we are least bothered about it. But I am explaining you how they have created all of this. Huh? How they have created all of this. This is not important. This is not important. But internally whenever I'm giving any call this is how it is happening. Okay. So that is why I said numpy is super fast. Do we say the operations of numpy are vectorzed in nature? Vectorzed vectorzed means the body is written in uh C and C++. Okay. Now, one last thing. Is that the only difference between a numpy array and a list? No. Also, one more thing. NumPy re arrays and arrays are collection of items of the same type. Very very important. List can have items of any data type but array compulsarily of the same type. Now how much is numpy important? Well, too much important uh just to show you. I'm just randomly opening few of the deep learning case studies, right? I'm just randomly opening few of the deep learning case studies and one machine learning case study as well. So if you see over here in this deep learning case study of image classification, I'm importing the numpy And what all functions of numpy have I considered using over here you see np new access np range then I say np where np.trandom random right so I may not be able to explain you what are we doing over here but I just wanted to show you that it is getting used np is what I have imported numpy as similarly over here as well you see npexpand dims np range np where nprandom similarly np.round round right then flatten function that's also of numpy np mean okay np log so this is just to make np dot array this is just to make you understand that c boss this is required ired. Uh this is just to make you understand that it is required and you know what see uh when till 12th standard in the examinations they never allowed us us means we the engineers to use a calculator in the exam but in the engineering when we took admission all of a sudden using calculators was allowed so at that time why didn't they expect me to do all the calculations by step why was a calculator allowed because calculator was mandatory at that time when I was on 12th standard. I was in 12th standard, 10th standard or anywhere below that right first, second, second, right up to 12th standard. Well, I was doing all the simple math. But now I am doing advanced math. So I am already assumed to be a pro over here. So when you are doing the advanced mathematics if you need something you use that go up and continue coding the advanced level. So at that time numpy is like this you know assume this entire thing to be like numpy the basics are over here. So whenever you do any machine learning stuff you need not implement all these small small things for that you can say numpy would you please solve my issue and it will do the needful for you right as simple as that. So coming back you know that was just to make you understand the practical significance of this topic. [clears throat] Okay. Okay. So now let us get into number. Let us start with numpy. So first thing is we will have to import the numpy library that's just import numpy. So import numpy as np. Now is it mandatory that you import it as np? No it is it is a de facto standard to label it as np but it is not mandatory. So but most of the people or rather than saying most almost everyone labels it as NP only. In fact if you go to Google and look into any of the documentation as well you will see that numpy library only in picture. So import numpy as nbp and then I would like to print the version of numpy that comes as np dot double version double underscore it's two times and done. Now if you go to collab maybe your version is different but that does not matter right you just need not worry about it whatever be the thing now let us create a onedimensional numpy array. So first thing is syntax np do. array you have to give the data and the d type. You have to give the data and you have to give the d type where data can be array like object and d type will be this and also d type is optional if not provided numpy will infer it by itself. So if it is optional I would like to first of all play around with it. So I say I want to create a numpy array. So I'm using the array function np dot array. And you can pass in a list or a tuple or a set and it will create a numpy array for you. But as I told you during the basics of Python that whenever you do this always 95% to 98% of the times you will be using list only. So I'll be using a list. So let's say if I say 1a 3a 5a 2a 8 these are the things you know. So I will print that array and also print the data type of that array. So print the data type of that array. Then after that you see this now how is a numpy ideally array is created. So numpy area look at this figure of 1 3 5 2 8 1 3 5 2 and 8. This is exactly same like list. Yes. 0 1 2 3 4. In fact, -1, -2, -3, -4, and -5 as well. As you all can see, this is a 1D array consisting of total five items. Yes. So, let's get into this. So here we have got this creation of an array and the data type is numpy nd array okay n dimensional array because it can be 2D 3D nd etc. Now uh after this creation of this I I want to show you how I can do indexing on this array because I see the indexes over here. So indexing well it is exactly same as list. So I don't want to invest a lot of time. Ar r of zero will be 1. A r of four will be 8 minus one would be 8 minus 3 would be five and so on. So a r of zero would be 1. A r of 1 would be three. A r of -1 would be 8 and minus2 would be two. And when I run it, you can actually see it in the same way. Slicing is also same as list. So if I say 0 col 2 then the starting index is equal to 0 and ending index is 3 -1 which is considered as 2. So whatever we studied in list everything is applicable over here. Okay. Okay. After this uh I come to the creation part. Okay. So this is what we created. I also want to tell you that overall the D type is what I never gave. Right. So this this was the DT type but I didn't give it that was the syntax we take it over here so that the code becomes clean. So here I have 13 528 and I didn't supply the D type but I can print the D type. So I can say print the name of the array is a ar r. So I can say a r dot d type. So what will it give int 64? If you are using the very very very very very very very old system it might show int 32 as well. It shows 64 or 32. We are least bordered. It is integer. Huh? Rest of things we never require. It's just saying over here that whenever I say int 64 inside this 64 is nothing but the bit integer. If you understand this technical thing well and good else it is just not important. If I supply over here one of the value as 2.5 then I said numpy needs all the values of the same data type. However, here you can see it is a float. So it's a clear case of error. Numpy says what I'll do is convert everything to float because float is a higher data type. Float is a higher data. So it converts everything to float. Okay, then I can actually create a numpy array with the same thing and I can use the DT type parameter. So D type if I say float then you can see it is of float type right and I I can I can also say float 64 by the way but not required right that's what I said right you write float or you don't mention it it will automatically figure out so you never mention it that's always good better you don't mention it there if I say integer. Obviously all the values are integer. So what is the point in me mentioning? But imagine if there is a value like this. Now if you say database is equal to integer, it will actually do type casting on 5.22. So 5.22 2 to become five because you forcefully asked it to get converted to this. Okay, that's one thing. Okay, so this was our 1D array. I want to say That was the 1D array. Now I am planning to create a 2D array. Something like this. Two rows, four columns having the values. It can be anything. This is the row 0, row 1, column 0, column 1, column 2, column 3. Let's call it as a ar r only. You know how do you define it? Every row is a list and this entire thing you have to put it in a bigger list. Every row is a list and this entire thing you have to put it into a bigger list. So here I say let us create a 2D array. So I will see over here how I create for 1D array I use one square bracket for 2D array there will be two square brackets if you recall this is how the figure was now there are two square brackets that's my row zero that's my row one each has to be given in the form of a list so it's built on the top of list only so I can say is equal to np array square bracket close it. So outer list created this outer bringing bracket inside this another list 10 comma 20 comma 30 comma 40 50 60 70 80 that's what we said now see how conveniently I have created it out trigger and same everything same see 1D array singles Single square bracket. So single square bracket 2D is two square bracket. It is not mandatory that you write it like this one below other. You can also write everything on a single line but this adds readability. So what I mean to say is it is completely okay if I write it like this. It's just that it is not that readable while creation. So I don't prefer it. I prefer this. You don't end up doing mistakes while the array is getting created. 10 20 30 40 50 60 70 80. It's a numpy array. How do I know there are two rows and four columns? There is a shape attribute. So if you remember that I can print a arr dot type and that gives me it's in 64 and if I say a ar r dot shape that will tell me how many number of rows and columns are there. So I want to tell you that D type and shape are attributes of the array and they are not the methods. So the method that we have seen is the array np dot array but d type do we have a round bracket after it like np array has a round bracket or here shape no these are attributes there are hardly five to seven attributes so there is nothing challenge as such in remembering them very very few attributes are there so right now don't ask me what all are the other attributes anyways we'll be studying them in detail so right now you just need not worry about it. I was saying it showed me that there are two rows and four columns. But but but but can I write the same code here in the 1D array? It's also called as a ar r. So it will once again declare a ar r as 1d array. And if I give shape then what happens? See here when they created the function the return type is a tpple. The return type is a tpple. So here also it will return a tuple only five comma indicating that there are five items in the array. There are five items in the array or five values in the array. This is how you create a 2D array. And here if I want to access number 20 it is at row 0 column 1. So indexing print a ar r r r r r r r r r r r r r r r r r r r r row 0 column 1 20. I can also write the same thing as a ar r of 0 comma 1. Both are same both are same. You can see both of them are giving the same output. Whichever you find easy, you can write it down. Okay. Now, negative indexing is also there but you need not do it, right? Um, it's also there but it's not required. How muchever is required, we can definitely look into it. So, in the same way other things we'll do it later. Let's not get into the complicated part of slicing right now. I now want to do one thing and that is I am planning to create one single array. Yeah. Uh let's assume this to be it can be any values. I'm just randomly assuming it. This was 1D array. This was 2D array. Now I aim to create a 3D array. Now the 3D array is not a true 3D array. So something like a three-dimensional object. No, nothing like that. It's a the it's called as a 3D array but it's not threedimensional. So you know how I say I say that they this entire thing is one array where this is the zero array and the lower is the first array. Inside the zero array, this is row 0, row 1, column 0, column 1, column 2. Inside the first array, this is row 0, row 1, column 0, column 1, and column 2. And the way I define it is something like this. First of all, every array is a list. So this is an array. I'll just mark it like this. So that's one. That's one. And you cover it by an outer one separated by comma. Done. That's one. That's one. And you cover it by an outer one separated by a comma. Done. And that's and that's separated by a comma. That is how it becomes a 3D array. say 1 D array one square bracket 2D array two square brackets 3D array three square brackets let's check it out whether that's really the case so I say is equal to uh okay it actually heard me you see over here now I can write everything on a single line as I told you but I am not writing And let's check it out. First of all, the way I have declared it exactly same like this pink brackets. Print ar has printed it exactly same like my figure. Inside this figure when I say print array entire thing type of it is a numpy nd array shape you know what is the shape trying to say it says that we have two arrays each of two rows and three columns. Perfect. That's perfect. It says I have two arrays each of 2 + 3. You can see that this is 2 + 3. This is 2 + 3. And when I want to access 60, 60 is in the zero array. Row 1 column 2 0 1 2 that gives me 16. Or I can write 0a 1 comma 2 1 02 is it 90 1 02 1 is this 0 is this and two is this 90 that's correct so that is how you create a read array just pasting this as an image for your future reference so that becomes little convenient for everyone to figure out. Okay. So that is how you create it. Now I want to say that I said when you look at 1D node now anything more about 3D we'll check it out later. Right? Right now it's not required. Right? Now this much is more than sufficient. This was the 1D array. I said that here by any chance if I put a float value and I print it and I check the D type of it. I print the type and I also check the G type of it. Then it is everything converted to float because between integer and float because between integer and float float seems to have been given a higher precedence. Therefore everything became float. If I put a boolean value like true and a false they are not allowed because array has to be collection of the values of the same data type. But if I put it then true gets converted to 1.0 and float to 0.0. No problem. You know what I am planning? I want to also show you some other functions. Right? So I want to add over here some basic functions of numpy precisely the arithmetic operations or not arithmetic operations arithmetic functions. So I want to find the sum of these values. How do I find out? Well, print np dot sum of this. So some of these values would be given over here. In fact uh you know before I do this you know before I do this let us do it on a basic array without a float without a float let's say let's keep it simple as well I think that would be one or let's say 10 comma 20 30 40 50 right so let's See so 10 20 30 40 50 is the 1D array which is created. I just say I want to check the d type everything is int. So it is int and the output is 10050. That's good. Let us try other functions. So I have the minimum I have the maximum. I have the average which is mean. I have the median. I have the standard deviation. I have the variance. Many functions are there. So I can say n3 dot minimum of the array which is 10 maximum which is 150 sorry 50 mean which is the average median which is the center value after you take all all of you aware what is the median after sorting the values the one which comes at the center right? Yes. After sorting the values, whatever comes at the center is nothing but the median. So, and then you have the standard deviation and variance. Uh, so we'll be learning this standard deviation and variance when we get into the statistics part. But in very simple words, standard deviation is how much you are deviating from the mean. For example, if I say for me to go from my home to office it takes 1 hour because the distance is 25 kilometers. However, if there is no traffic, I'm lucky. I end up reaching in 45 minutes, 15 minutes earlier. And if there is heavy traffic, I end up reaching in 1 hour 15 minutes. So, the average time taken is 1 hour average, which is the mean. And the standard deviation is 15 minutes. So I take mean plus or minus one standard deviation to reach my office. So it can be 1 hour - 15 minutes which is 45 minutes to 1 hour + 15 minutes which is this. So that is the range which I take to reach my office. So this can help me to take a decision if I have to go to my office and if I have a meeting let's say with the manager then I will end up considering this time. So if I have a meeting I'll prefer to leave 1 hour 15 minutes early because I cannot be late and if it's a regular day and I can take that risk I can assume 1 hour or 45 minutes to reach the office. So accordingly I can take that decision as well. Okay. So that is nothing but the standard deviation. So what am I doing over here is just take out this as well and yeah you have it over here. Hm. So standard deviation and variance. So standard deviation is nothing but the square root of variance. So 200 and square root of this is 40.4. Okay. You also have the product. What will the product do? 10 into 20 into 30 into 40 into 50. You also have the cumulative sum. What is cumulative sum doing? Well, cumulative sum is saying what is 10 + 20? I here. Yeah. So, first it says 10. 10 + 20 is 30. Then this 30 + 30 is 60. 60 + 40 is 100. 100 + 50 is 150. So this is happening with respect to the plus. If I use compro then it will be like let's take this 10 down. What is 10 into 20? 200. What is 200 into 30? 6,000. What is 600 into 40? 24,000. 24,000 into 50 and so on. Okay. Difference. Now what is this difference doing? So as you can see difference it is finding the difference between consecutive values 20 - 10 10 30 - 20 10 10 and so on np non zero now I I don't know actually what is this function it's suggested but n it's never never required in my career ever np do non zero and What is it saying? So I don't know. It suggested me. So what will I do? You know, let's say I don't know. But it is suggesting me something. So copy this. Go to Google. And first thing what I want to show you is the numpy official documentation which is available on numpy.org website. It says this version is released. I can click on that. Okay, this is what is the one I can learn dumpy quick start tutorial and lot of functions lot of help available over here to let me see shape size all these things I'm going to first of all come and just put in the documentation link here second thing np nonzero I don't understand it so I'll just search np.nz non zero and I'll also open up this geek forge geek link and this link because you never know sometimes you don't understand from the documentation nowadays I prefer using chat GPD for getting this answers but you know I'm showing you approaches so this will return the indices of the elements that are non zero okay so there is a example as So indexes of the elements which are non zero 0 comma one. Where is this now? Ah, see easy. 0 1 this is row 0, row one, row two, column 0, column 1, column 2. So 0 and 1 is 8. 0 and 1. 1 and 0 1 and 0 is 7 2 and 0 - 5 and 2 and 2 should be the last one. That is what it is written. Simple. So in this way if you get up get stuck up somewhere there are a lot of resources available as help nothing to worry about. Anyways so I can just execute that and none of the values are zero. So obviously it will not give me a zero anywhere. Okay. After that sort, it's already sorted. So can't do anything. Arc sort. Now what is this arc sort? Well, uh that's little difficult to say at the moment, but I'll explain. See right now the minimum value in this array. If you see I'm talking about this argument what is the minimum value 10. What is the argument of that zero arg max? What is the maximum value? 15. What is the argument of that four? Then arc sort is sorting as per the argument. So I'll do one thing for this arg. We will see. Okay, it's not sorted. Now I say empty dot Argument argument would be 20 sorry minus 30 which is at index 2 arg max and ar I'll just tell this now here and here this is the smallest value it is at index two argument highest value is 101 which is at index one sorry index zero and if I have to sort it in the ascending order the ascending order will be minus 30 20 40 50 1 argument y + 30 is 2 20 is uh 1 40 is 3 then 4 and then zero same thing now that's arc sort what will be the argument if the array is sorted what will be the arguments which will be coming if the array is sorted so I'll only take the common functions now so we saw that These work. Okay, we saw that these work. Now here when I am taking one item float, do this still work? They work. Okay, they work. The value which I changed is this became 2.2. That's fine. When I convert this sorry add a boolean value you see over here boolean values true and false become one and zero and the d type is still float only so float is having higher precedence as compared to integer the boolean I'll do one thing Now let me add true and false. So we know that nothing happens. It will be float only. But what if I add a string like dan? Do you see uh uh uh what happened? So why is oh I wrote it in the same thing or what? Oh right. So we have this and added dion. So see everything became string. Oh that means float value did not convert everything to float that also became string which indicates that the highest precedence is string. String has the highest precedence. Okay. Okay. Fine. That's understood. And if I uncomment this from cell number 10 tellwards you get this. It says that here only right on line number 10 it says I cannot find the total. Obviously how do you find out the total of these string values? They are not integer. Everything is converted to string and numpy call that unic code 32 and that is important if you recolct what did I say numpy is mainly used for what is the full form of numpy what is the full form of num so it is only used for numerical calculations you cannot do string calculations so it is very stupid if you try out any of the numerical computations. Right? So over here U stands for uni code. Uni code is nothing but a string representation format. If you are not aware of that uni code number system this is how it was 11 this is how it represents everything. So everything is represented by something right as per uni code level format. So that's okay. That is done. Then I say images Abraham Lincoln. Whenever you represent images, images are represented something like this. Movies are represented something like this. So as a human this is how you see image but a machine sees it like this. Combined it looks like this. As per a machine number zero means black, number 255 is white. So you see over here these are pure black and they are 0 0 or you can see over here third last row 255. Then what happens? You know because all these values are between all the values are between 0 to 255. What will happen if I perform the operation of dividing by 255? So what will happen when 0 is divided by 255? 0. What will happen when 255 is divided by 255? 1. Now answer one question. Do you all agree that any number between 0 to 255 when divided by 255 will compulsorily be in the range 0 to 1? Do you all agree that any number between 0 to 255 when divided by 255 will compulsorily be in the range 0 to 1? Yes. This is what is called as normalization. That will be helpful in machine learning and deep learning. But this is called as normalization. And why did I all of a sudden go out of the topic and started explaining this weird thing is because it has some significance. Now so like because it has some significance I thought of explaining this to you. The next function which I want to talk about is NP.0 np.0 zeros. It says that if you give me five, print it. I am going to create an array of five values all zeros. If you give me 12, a numpy array of 12 values, all zero. You can also have the DT type facility to be chosen. So if you say DT type integer instead of all the float values like here it will choose as integer. Okay. Now it says that dash if he give me va 5 then all the I'll be creating a matrix of three rows and five columns all zeros and this 3 can even be supplied as a tuple it's not common but you can supply it so that both of these are ultimately going to create a numpy array only if you want I'll just print the type of array. It's a numpy ind. And here as well, it's a numpy ind. It's not that because a list is given or a tuple is given something else is getting created. It's absolutely the same. Okay. So here as well I have the facility to give d type. It will create a numpy array of three rows and five columns. As you see over here, it will create a numpy array of three rows and five columns. Okay, that is what we see. Well, uh what I wanted to say. Yes, just like this zero, you also have once function which will create an array of ones. And just like this once you also have twos functions, threes, fours, fives, by the way, which is wrong. I'm kidding. You don't have twos, threes, fours, fives. You only have zeros and ones. All the functions of numpy are added for some purpose. They are not added for time pass. These functions are not added for time pass. they are added for some reason. As I showed you, whenever images which are usually in this range are normalized, they become in this range. And that's why only zeros and ones exist. Twos, threes, fours doesn't exist because sometimes in image processing, what happens is let's say you are working on some let's say Instagram, right? You upload some image to Instagram. You know this is let's say the image that you upload to Instagram and you are trying to apply some filter on it. So Instagram will be creating a copy of this image. How is a copy created? The pixel values cannot be copied directly. First you create a dummy matrix of all zeros and then one by one as and when you are applying let's say a blur filter. So what will happen with that pixel? That operation will be replaced over here. So that zero will be changed. What happens with this pixel? That new pixel value is saved over here. So while creating a copy it can be a matrix of all zeros of the same dimension that you create or could be one. And what is why these functions exist? As I told you all of them exist for some reason, right? So these functions are mainly useful while the deep learning stuff that was the zeros and the ones function. Now can someone explain me in simple words what is an identity matrix? What are the properties of identity matrix? Uh okay, identity matrix is here and in identity matrix as well of course you have that D type. D type is almost there everywhere. So you see over here you have the D type and there is a I function as well which also creates an identity matrix. I can show you NP do I of three. Now you will be like but uh how is this different? Because if I copy this and if I go over here then it's absolutely same. Why? Then what is the difference between the identity and the I function? Well here we have the K. Here we have the K. I I'll do one thing. So I'll create a identity matrix of pi + 5. Okay. So this is how it looks like. So let us let us let us let us let us create it and pi cross pi. All right. Oh, row 0 1 2 3 4 column 0 1 2 3 4. By the way, that's K. Take can take any of these values. How? Well, copying this, there is a parameter which is called as K. If I take K=0, if I take K=0, no change in the output because K 0. same same like this figure. But when I say a as one now see what happens. The diagonal the diagonal is here. Of course it's not a perfect identity matrix and I am not calling it as identity matrix either. Okay. When k is equal to 2, see it shifted here. So from here 11 1 1 started. When k is = 3, it shifted here. And when k is = 4, it shifted here. And when k is = 5, all zeros. and when K is = 6 all zeros but you know we don't need it that often but it just goes in complement with this so see all these functions exist for some reason but this is very very rarely required the I function uh it's not required only I mean it is applicable for positive values only uh I I don't know. I never tried it to be very very honest and there is no need to try as well. But let's say if I say k is equal to minus1 does it start from back or not? I don't know but okay it is minus one means it is taking the column wise row wise I never tried it. First time you asked me and I'm trying it out. Yes I think but but but not required to be very honest. This is very very rarely rarely required. Right. So don't worry about it. I just wanted to teach you identity function because I function goes complimentary to that I showed you but it's not required. Okay. So don't worry about it at all. But I'm sure and sep you got the answers to your question. Am I correct? Cool. Now see we have seen so many functions inside our numpy. I don't even remember what all functions we studied. Let's say sum function main function sort function argument main function. You studied the arg max I on and on so many functions and I say that we all studied this inside the module. In fact, there are some other functions as well which I would like to talk about like a rand function, rand function, rand function. All of these rand rand rand r r r r r r r r r r r r r r r r r r r rand r r r r r r r r r r r r r r r r r r r rand r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r rand stands for random and because these functions are more or less achieving generation of some random numbers they are also dumped into a module which is called as random and the best part is they are a submodule inside numpy if you have to call let's say the rand function then you say np dot dot random dot ran that is how you call this function. I just wanted to show you that okay this is how it is done. So I will just get this things for you. Now we don't have to look into all of these functions but you saw in the deep learning notebook that random module is little used not too much but little used. One of the perfect application of random module is it can be used for generating OTPs right every time you're doing some transaction some OTP has to be generated which has to be purely random not following any pattern and that should go on to the user and then it must tally so how do I use it well np dot random dot rand end of five. So it will generate random numbers between 0 to 1. Five random numbers. Every time I'm running it is generating five random numbers between 0 to 1. Then you have random. It will generate the numbers between minus1 to one. See every time I'm running you have rand you specify the value and it will generate the numbers in between them low and high. So between 1 and 10 generate fine numbers. It will always be between 1 and 10. Look into the last line output. Between 1 and 10 fine numbers will be generated and many are there. So random model is responsible for generating any value between zero and one. Yes, very similar to the above one. I do agree and so on. We need not look into all but yes in deep learning as well when you have a set of mclassifications and if you want to randomly plot one of the mclassified image that random model turns to be of very heavy help. Okay. Another thing uh something very common in the machine learning like I told you sometime back code is not important right now right in machine learning let's say when you have a data set you always have to separate what you have to predict from what are used to predict like the pink values are used to predict the values in the yellow box. So you have to separate. So now what am I going to consider is an similar analogy where I have values 11 22 33 44 55 66. So I want to tell you that usually usually like here the tips column is the second column but usually the column to be predicted is the last one and the ones which are used for predicting it are here. So what I want to do is I want to separate it out this blue part into X and this pink part into Y. And before that I would like to write down some naming convention. This is row 0 1 and 2 column 0 column 1 column 2. I can also call it as -1 column, -2 column and minus3 column. Same row also can be called as -1, -2, -3. H let us create a data set. So I say np dot array Perfect. This saves so much of my time. Now how I do it is going to be very very interesting. See I say in my X array which rows I want and which columns I want. So I say I want all the rows that is from row 0 to row two. So I want it from 0 to two. So I will write 0 column 3. and columns. I say I want all columns from C 0 to C1 for which I will write 0 col 2 then only it will take column 0 and 1. Okay, I can also write this as just colon because by default if I don't give the starting and the ending index then it takes everything and I can also write this as I want it to start from zero and go up to minus1. So when I say col minus1 it's by default zero and when you say minus1 that means it will go from 0 to -1 -1 which is 0 to -2 column 0 to column -2 that part. Hey and As far as Y is concerned, I would say I want all the rows of column minus one. So these are my rows and this is my column single column which are single column. I didn't write colon minus one. Colon means start from zero. I am directly saying minus one. That means clearly I'm saying I want only minus one column. Let us check it out once. So, x is equal to array. You can see over here, right? That's X and that's Y. Have a close look at it. It's not difficult. It's just different. I will also first of all like to add this image so that this doesn't go and then you just quickly keep on observing this. I repeat my X's all the rows that means row 0 till this. So I can say 0 col 3 because row 0 to row 3 which is going to take 0 1 2 and column is 0 to my 0 to 1. So I will I cannot write 0 to 1. If I write one then it will go up to zero only. So I will say 0 to two. So it will go from 0 to column 1 column 0 to column 1. And this 0 to three can be conveniently just written as colon only because 0ero can be skipped and I can only write three. 0 is optional because by default it is zero. And here also zero that can be written as just colon 2. I can write it as colon 2. Right? And 2 is nothing but minus one. Now 2 is nothing but minus1. So I can also write it as minus1. And that's what I wrote. Colon minus1. And here when I say colon I mean to say I want 0 to three which is as good as saying everything. And here specifically I can I can write I want column 2 or I can just write 0 col 3 and I want the minus one column. So ultimately it's one and the same. So now that we are uh done with this the next function which is again very popularly used in machine learning, deep learning and almost everywhere that's going to be your a range function. So first of all if you recall the range function where I used to say for i in a range of five print i that used to print 0 1 2 3 4. Numpy says I also have a same function called as a range. Absolutely same function called as a range. So a range will work same like range function but the difference is a range would be storing these values into a numpy array. So here I say np do arange file and if I let's say save it into a variable like a ar r r then you can see over here it has this you know I can also print the type you can see this 0 1 2 3 4 it's a numpy array this 0 1 2 3 4 it's a numpy array in that even if I say np dot in range of 12 0 to 11 np dot in range of 1 to 12 it goes from 1 to 11 if I say 1 to 12 at a step size of 2 it goes like this exactly same like a range the difference is it will create a numpy array. That is how this is going to be different. The most important thing over here is the output will always be a numpy array. The next thing which I want to talk about is if I say nparange of 12 there is one more function called as reshape. This reshape says I will reshape this array. Reshape says you have the value 0 to 11. Right? Using np range further using reshape I will reshape this into three rows four columns 0 1 2 3 4 5 6 7 8 9 10 11 okay also for this thing to work 3 into 4 must be equal to 12 if and only if the multiplication of this turns out to be 12. This is possible else it's an error. Let's try the various scenarios. I can say 4a 3 and it works. Four row three columns. I can say 6a 2 6 rows two columns. I can say 2a 6 works. I can say 1 into 12 is also 12 right. So obviously now the thing is what is the difference because np range and np range and both of these are giving an output that's the first output that's the second output. Can you tell me what is the difference in the two or they are the same? What do you think? Of course they look the same. What do you say? Very nice. Very nice. Correct. This is onedimensional array, single square bracket. And this is twodimensional array. Superb. Very good. Very good. 1D array and the 2D array. That is absolutely fine. Okay. Now this reshape I said whatever you put over here must the multiplication must be equal to 12. So 3 into 2 is 6 into 2 is 12. 3 into 2 into 2 is also 12. It does create a three-dimensional array. You can see it has created three arrays each of size 2 + 2. So if the multiplication is equal to this it works in the same way. If you want you can create here 2a 2a 3 3 2 are 6 6 2 are 12. You see over here two arrays of two rows, three columns and now you see over here this is four dimensional array four brackets. You never require it at least I never required it ever ever ever but this is just for your conceptual understanding that okay a 4D array is also possible. Okay. So, next thing after this is uh if you have a numpire. Now, this is uh I can call it as advanced slicing. Assume that I have a Numpy array and the values are 0 1 2 3 4 5 6 7 8. Now in this numpy array um if I say bring a r of control a control s let's put it here only print a r r Oh, this is giving the output as this. Okay. See these are the rows and the column indexes which are selected. Row 0, column 1, 1. Row 1, column 0 6 3 row 2 column 0 6. Done. Now this is [clears throat] done. Now we will never use such kind of advanced indexing but this is one way of doing the indexing. It is not mandatory that you have to use it. I never use it in real life but if required it can be used as simple as that. Okay. The next is I want to talk about the concept of broadcasting. This is again a very important concept broadcasting and broadcasting is a powerful mechanism that allows the numpy to work with arrays of different shapes when performing the arithmetic operation. What I mean by that, we will understand it in the code. But this is required in machine learning and deep learning for handling matrix operations when the arrays are of different dimensions. You have a smaller array and you want to multiply it with a bigger array or the vice versa. These operations can be done over here. So let's understand it with the help of a very basic example. If I say np do range of 15 and reshape it to three rows and five columns. If I say ar r + 5 mass teacher is going to give a tight slap on my face saying that dan you don't know a single value cannot be added to an array. Two arrays can be added but a single number cannot be added to an array. But numpy says don't worry I can do that. It says what do you want to add? I say I want to add the number five. The dimension of this is 3 + 5. No worries I will broadcast this five. So this has become three. Now it is 2 + sorry 3 + 2 3 3 + 3 3 + 4 3 3 + 5 done. Now I'll perform the addition and there you see the result. So what has happened over here is exactly what we call as broadcasting. Just give me a moment. Now we say that's broadcasting then in the same way a r minus 2 also works. A r multiplied by 2 also works. So every value will be multiplied. You can just check out a ar r and you can see every number is multiplied by two. Now if I have an array np a range of 15 and resetting it to 5a 3 and I have another array npa range of five and reshaping it to 5a one and I say now let's understand this that's an important thing so you are saying that array 1 and array 2 is to be added this is array 1 this is of + 3 you want this to be added with this array 2 which is 5 + 1 which is 0 1 2 3 4. Is that possible? No. Because it needs 5 + 3. It will broadcast this broadcasted one time. Broadcasted one more time. This also become 5 + 3. Corresponding values will be added. Like you can see this zero and this zero are added. answer is zero or let's say for example this 13 and this four are added this 14 and this four are added and you can verify few of the answers and that's how you got it so reshaping that doesn't mean any reshape will happen first you understand this then I'll tell you the other What? See anything cannot be reset. For example, this works. But if I have instead of 5A 1A 5, then see it doesn't work because 1a 5 means you are saying that to this you want to add this even if you broadcast it. So this is 1 comma 5. Even if you broadcast it let's say 1 2 3 4 5 times it become 5a 5 right so this is 5a 3 after broadcasting this 1a 5 it become 5a 5 still the dimensions are not same so will you extend the dimensions of this no it's not possible so everything will not work only feasible operations will work and they are useful in tensor multiplications in deep learning So anything cannot be multiplied with anything. That's an important thing which I wanted you all to know. Okay. The next is I want to go ahead with linear algebra. So linear algebra is the integral part of machine learning. And let me tell you that your numpy was initially developed for linear algebra and matrix operations only. However, slowly and steadily it translated to the machine learning part. So how do I get started with this? Well, let's do something which is called as matrix multiplication. So how do you consider multiplication of the two matrices? Let's say you have np dot array 1 2 3 4 5 6. You have another array. Yes. First of all, what is the order for matrix multiplication? Well, a matrix A of order M cross P can get multiplied with P cross N to give the order M cross N. Right? So, the columns of first matrix and the rows of second matrix must match. And the way I can perform matrix multiplication is dot product printing C or there is another way to perform matrix multiplication and both of these are valid. And the third one also you can see all of them are giving the same result. Either you use the dot function and supply the arrays or you use at the rate in between the two or you use the mat. All of them are giving absolutely same output. You can check it out. Okay. Then oh remind me one thing. I have to show you the speed of the numpy that I forgot. Then after that I say transpose. If you want to find the transpose of the array. Let's say I have a numpy array and I'm printing this. That's the quick way I can find out the transpose of the array. Right? You can see what is transpose. Basically 1 2 3 1 2 3 rows became columns another row became another column. Same happens over here. So a t that's a quick way. That's not a numpy function. Numpy function is np dot. There is one more function which is a transpose. Again this is not a numpy function. Wherever you see np dot only that is a numpy function. So though though many these many are there you can see the output of this output of this and this are the same. But of course you need to understand this is the only numpy function numpy offer function to do this. Then the next thing which I want to do is solve linear equation. So if I say consider the following system of equations. I say 7 x + 5 y - 3 z = 16. Then I say 3x - 5 y + 2 z is equal to - 8. And then I said 5x + v y - 7 z is equal to z. Find the value of x y and z. We have done this thing in school days multiple times. So we have done this thing in school multiple times. So control arr chat gpd and I go over here and I ask it to solve it. So 1 32 these are the outputs. Okay. How will numpy solve it? Numpy says that what you can do is these all values you save it into an array let's say a and you know how will I save it as a two-dimensional array these all value you save it into another array called as B and then you just say not nol a numpy has a subm module linear algebra where you solve a and b and it will give you the answer. So I say a is equal to this b is equal to this and I say np dots solve and you can see I'm getting the output over here as expected. So how are these values set up? Well, if you observe 7 5 - 3 first list 3 - 5 2 second list 53 - 7 third list they are stored as a 2D array 16 - 8 0 1 array and then linear algebra solve and you get it end of the story. Let's go ahead. Let us get into do you remember something which is called as a determinant of a matrix you might have studied during the school days. Yes. If someone has forgotten the mathematics of how the determinant of a matrix is calculated then I cannot teach you that right now but I can give you the link to study it. So if I have to calculate the determinant of this matrix, it is always the 3 into 6 minus 8 into 4 - 14. talking about determinant of a matrix and I say a is equal to np and the way I find the determinant is simply numpy linear algebra modules deed function another way is determinant of a transpose but I won't recommend that I'm not even sharing that that's the determinant And then the next is inverse of a matrix. In the same way you might have studied inverse of a matrix. If you do not recollected, this is where you can study. It's very nicely explained. Just as an example, I can show you. So if you have this matrix ABC you want to find the inverse of it then it is 1 upon a d minus bc that is one upon the determinant of this matrix and then this diagonal elements a and d are swat and the off diagonal elements are multiplied by minus one and that is the determinant that's the inverse so I Okay. So let's say you're having a numpy array and you directly use the linear algebra inverse function to do it straightforward. Do you know what is the trace of a matrix? Do you know what is the trace of the matrix? expecting you know what is the trace of a matrix. Yes, very good. It's the sum of the diagonal elements. That's perfect. Trace of the matrix is nothing but sum of the diagonal elements. So I create an array and I say np trace. So 1 + 4 is five. I have the trace ready. I have a trace ready. So that is nothing but the trace of a matrix. Then now the next one are the few important functions which I would like to say. Uh so I say uh the next functions which are left out As per we're almost done just few functions which are left out. The first one is the line space function. Now what is this line space function? So line space function will be generating all the values in uh uniformly uniformly for example for example if I say np.space time space. You know what will it do? Between 0 to 1, it will be generating five numbers all equally spaced. See 2.5 - 0 uh 0.25 - 0 0.25.5 -25 0.25. This minus this 0.25 this 0.25 25 if I say that generate seven numbers.16 - 0.16.33 -.16.16 1 - 83 again 6 only. Okay. So that is what is the lin space function. The next function which is used is the empty function. So here so what does the empty function do? Very very simple. If I am saying empty 2, 3, what is it doing? Well, it will be creating a numpy array without initializing the entry. So, array will be containing any arbitrary uninitialized values, right? So, that is the main idea behind it. Now, you'll be thinking where is this even required? Well, see empty function is required in high performance computing and the large scale machine learning pipeline because when you you know empty will be useful when you will need to fill the array afterwards avoiding redundant zeroing. So those all are are the places where empty function is useful. In deep learning also this is useful when you uh quickly want to allocate the memory during the mini batch gradient descent technique. It's difficult to say right now but yeah I mean that's what I can say. The next thing is the flatten function. So what does the flatten function do? It will flatten the array. It is very popularly used in image processing. If you look, I say array is 1 2 3 for it's a 2D array. You know how it is visualized. You can you visualize it usually like this. But when I flatten it, see this is 2D. And after flattening it looks 1D. That is the difference. Then second you have np dot ravel. So ravel is also going to be flattening the 1D array whenever possible. That means it will always avoid copying the data unless necessary. So here with respect to the Python example, you will see that this also does the same thing. You have the array and it is flattening it only. But the difference is unlike flatten it will always return a copy. So ra will return a view if possible making it more memory efficient. Then very simple functions but they are used in machine learning. The next is swap access. So what does swap access do? Let me show you this. Have a look at this first. Uh I have this array. One array of two rows and two columns. Original shape. one array of two rows and two columns. I am saying I want to swap the axis of this arrays and 0 comma 2. You can see I got two arrays. Each of this comma this will do one thing. Let us print the swapped array as well. So you see it just changed the dimension and I'll also print the original array. 1 2 3 4 was printed as 1 3 2 4. Now is that really very helpful? Well, no. Very very rarely used. In fact, uh I don't remember me using it anytime in my work in last 6 years at least. So you can just consider it as an additional one. That's not really very very important. The next one which is important is the horizontal stack and vertical stack. These are required. So let us start with the edge stack function. So what does this edge stack function do? I'll give you a very very simple example. Let's say I have this array or or or let me say See I have this two arrays as you can see and horizontal stack has taken 1 2 and three for horizontally. And similarly if I go ahead with restack it has taken 1 2 3 4 vertically. See it converted these two arrays into two dimensional arrays each of this uh no there is a difference. Now platin and this although working wise they are same but horizontal stack can take up any number of values and stack it. Platin takes a 2D and converts to 1D but it can take this this h stack can take any number of array and convert it into one flatten list. Flatten will only take one array and convert it into this. Right? Yes. So horizontal stack and vertical stack these are the ones after that I guess uh I think we are done. Uh one thing which I showed you in the deep learning that I didn't uh cover restful things are good and the last function to discuss is the wear function. So what is this where function? It will be used in deep learning as I told you. So what does it do? Well, look into this code. 10, 20, 30, 40, 50 is the array. I say np dot where the value is greater than 25. It tells the indexes. These are the indexes two and three. where the number is greater than 25 or if I create two arrays and I say condition x greater than two. So here in this case over here in this case condition x comma y so you can see all the values in x greater I'll show you what is condition right let's print condition then you'll understand it x greater than two so these are the values which are greater than to three and four it give true and true so three and four Right. So index this. So now false false true true. So in this it will be mapping 10 is false 20 is false 30 and 40 are true. Therefore when I say np do where condition x right which is this and y and I am looking for the results. I'm looking for the condition x is greater than this. So x greater than this has given me what? X greater than this has given me the three and four and that is what is here and y has come as it is. But again you know uh more than this the first application is more helpful. This can actually little confuse you. This is more easy and more helpful. This completes this completes the numpy module. Now out of these functions which are the most common ones that we'll be using in day in day out there are too many for this. you are finding these functions too many but according to me these are the only functions that you'll be requiring it in a long run and if you internally look into numpy numpy is a module which you can spend an entire month to read over here you see over Here numpy can be used for quantum computing. We did not cover this part statistical computing. Neither we did covered this part. It is used for signal processing, image processing and so many domains. Surprisingly geoscience, geographic processing and so on. So we don't need all of this bioinformatics theorem etc etc etc right so you don't need so much of this is what you need to understand pandas right so that is the agenda for our today's session. Let's get started and we are getting started with the topic of pandas. So, pandas is an important library for data manipulation. That means you can change the data, manipulate the data. We say data wrangling which means cleaning the data and analysis. It provides or I can say the functions of pandas are as intuitive and easy to use as the data structure it provides. Pandas is built on the top of numpad right so it will work seamlessly and one of the most widely used library anywhere in the data science field okay it is used in each of the fields that's okay more importantly it is helpful in machine learning and deep learning now I come on to the data structures of pandas. So pandas provides two main data structures. The first one is going to be a series. And what is series? Series is onedimensional array. Yeah, we can see array very similar to an array only. See, this is one dimensional and there is also an access label. As you can see, it is very similar to a list or a dictionary. Okay. And how to do that? We will see it later. And data frame is a tabular one. So this is how it is. You can read about it later. But the most important thing is the data frame only. The most important thing is the data frame. Okay. Now so many things all of these operations can be done on data frame. Now as and when we go ahead we will be anyways implementing this operation. So let's not unnecessary uh invest lot of time on it. Okay. But all of these operations which are suggested are perfect. Now if I explain it to you right now there will be lot of questions because it recommended I just took it all the information was good but let's do it in action only. So now series and I like to look into the syntax for creating a series. PD dot series index is equal to index. So data is the data that I want to save and index is the index. Now if you not provide it is integer by default and it returns a series. Now obviously it's better I explain but before that we have to import pandas. Do not focus a lot on the content which is generated and you try to read it. Whatever I explain focus on that. So this will be printing the version of pandas after importing pandas. If you are having a different version then also it's okay. Okay. Okay. So this is one. So this kind of recommendations that we I'm getting over here which can be helpful for you when you study. Let's get into the process of creating a series. So first and foremost you know I'll keep it very simple. Let us say I am having a dictionary. So I have a population dictionary where I have California's population, population of Texas randomly taking the values, Florida's population, New York's population, Pennsylvania population, Illinois's population, and This I am printing it and I am also printing the type. So it has to be a dictionary, right? In fact, I need not even write this. So I will just keep it simple. I printed the dictionary and I printed the type of it. Simple, a very very simple dictionary and the type of it. Now if I say if I say over here but population is equal to PD do series and here I'm giving the population dictionary. So what am I doing? I am passing the dictionary to the series. We know dictionary is a key value pair California population Texas population. If I do this let's see how it looks like. So this is how a series looks like. Later on it will be required. So better make a note of it. So the keys of the dictionary are the keys in the series and the values are the values in the series. At the bottom you see it also prints the t type. Okay then. So always a series will look like this. Always a series will be looking like this. And the data type is pandas.core. series dots series as simple as that. Okay. Now let us get into the process of creating a very basic series. So I was talking about this as the syntax. Okay. I'll only take up this now. What I say over here is S is equal to PD dot series and let's say I give the numbers as 2A 6A 4A 8A 10 I did not give any index and I say print the series and print the type of it. So you can see the default indexes are being given. You can see the default indexes are being given here. What to understand is if you just provide a list it creates a series. In the same way you even provide a tpple it will end up creating a series. So don't provide tpple. Please don't provide it. You can even provide a set but you will not provide it. That's number one. Okay. So here what do we see? Because index is not provided. Default indexes are 012. In dictionary when I supply dictionary the keys became over here. Okay that's the very first thing. Now here you know how is this series saved? Exactly like list your indexing is also exactly like list but I am not talking about indexing at all. So if I say s of zero because it's a series s of zero should be two. it is. So now we saw that how I can conveniently [clears throat] create a series but in the series I can even pass in the index. So if you see carefully in the series I have passed in the index and now comes a little bit confusion that's why I didn't discuss now I should write s of a or s of zero to solve that thing I said we will be discussing this topic in detail right maybe s of a works but there are some other caveats to it we will discuss it later okay you saw over here that I can do it if If I say S is equal to two just a single value and printing S and printing the type of S. You can see over here a single item is also assigned. If I say S is equal to just a single number. I did not pass a list just a single number. then also it converts it. If I say s is equal to single number and then I pass an index, it works. That is also obvious. If I say S is equal to single number but I am supplying A B CDE E that's a clear case of error am I correct or not this is absolutely a clear case of error yes or no what do you think quite obvious right because one value and you are giving multiple things but that does not happen but that does not happen let me just make you recolct effect. Do you remember yesterday in numpy we looked at the concept of broadcasting with the help of which there was some numpy array which was two-dimensional and I added number five to it and the number five was added to each and every value of the matrix. Do you recolct? Yes. Because pandas is built on the top of numpy here also it doesn't let us down. It doesn't let us down. It broadcast the value as many times as the number of indexes. Okay. Uh so that's just I wanted you all to know that 95% of the times you will use data frame only. CV will be only required this much this much. But whatever is required at least we should know. So I'm covering all those things for you. Else CDs are very very very very less required. Okay. If I say I am uh creating a series by passing in a dictionary maybe something like this. 2 colon a then 1 colon b and 3 col c let's say that's it. So we know that you know keys and values whatever are being there they will be coming down. Oh what happened? Uh what happened? uh yeah round brackets close this is something which I'm not finding any interesting but but but but but in this same here after this if I'm giving the index and if I specify the index as 1 2 3 because here while defining I give 2 1 3 but if I specify the index as 1 2 3 does it work does it give an error well it will work and it will give more priority to the index order given and therefore it becomes as 1 2 3 important point to be highlighted. You can understand the difference between this and this Okay. Does indexes needs to be specifically written? It depends. Depends on your application. You may need to write it. You may not need to write it but ideally in a realtime project you will never create a series by yourself. Neither will you create a data frame by yourself. So practically the answer to your question is no. We never create a series only. Forget about indexes. We never create series. also arranging the also arranging the values along with the indexes that's correct anyways showing in the ascending order no no not ascending order showing it in the order which I specify in the index so what I mean over here if instead of this if I just say 2a three. So it will only select two and three. So it will select 2 A and 3 C. Now it is selecting whatever I'm asking it, right? Yes. Whatever I ask, it will select in that order. There's no ascending or descending or order to it. That's it. Now let us take a step ahead. Now I go on to data frames. Series are all one-dimensional. Data frames are two-dimensional. So syntax is this. whenever it's coming suggesting good good okay now I remember before that I having a population series created yeah you remember this right so this was the series that we created now I want to tell you that a list is used for building A series a series is used for building a data frame list series data frame. So I am planning to take up this series of population and directly pass it over here. So I say Z is equal to PD dot data frame and I only pass the data only pass the data and then I'll show you how the Z looks like. Yes. So it is converted to the data frame. In fact, just like yesterday, I can actually print the shape of the data by saying print Z dot shape. You see over here it says that there are six rows and one columns which means it is two-dimensional in nature. Which means it is two-dimensional in nature. Here here let's say here we had the series and I want to print the shape of the series and if I say population dot shape this is 6 comma 1 dimensional one dimensional observe over here this is one dimensional and this is two dimensional now we will be creating a data frame. Now again as I told sometime earlier you will never be creating a data frame from the scratch in real life you'll be reading the data given by the client you will not be creating it but in our session we learn how to create it as well maybe for working with some toy data sets you need it. So I say let us create a student data frame and I'll copy the syntax. So I am in the process of creating a student data frame. So here we see. Now first of all you just have a look at the output what we created that will give you a perfect clarity on what we are trying to build. Okay. And how is it created? Let's focus on that. So you see I have a simple dictionary. It's a simple dictionary. Key value pair key value pair. Keys are the names of the columns and the values are list. So whatever are the values that are the contents of the columns. Now I want you to follow this. Now this dictionary when passed into the data frame it creates a data frame. I always advise people to print the data frame directly as I showed in line number 11 as compared to what I showed over here in the print statement. See the formatting is way more different and this looks much better. Now have a look at the code. It's really very simple. Perfect. Now, it's better that you don't write print. It's always better. I mean, I always advise obviously this looks good, right? It's better not to write it. and and and when it is uh I'll show you now that should not come up with any demands but see here it looks more better in the white mode but it hurts the eyes right so yes this is what we have right so [clears throat] without print only is always good so that This is how you can quickly create in a data frame. Okay. Now that uh you understand how one can create in a data frame. Uh I can also create it the raw way. So what will I do is you know I'm going to import numpy and I am creating a data frame. PD dot dataf frame. I need to give the following things. data numpy.random.trand. So it will be responsible for creating what a five row three column matrix. If you don't recollect it, let me show you separately. It will create a numpy array of five rows, three columns. Right? I'm running it again and again. You can see it's creating a numpy array of five rows, three columns with the random values. Huh? Okay. ran it so many times from 25 I went till 69. So I ran it more than uh almost 35 times. So then the second parameter if you recall the syntax right that was the syntax. So the next parameter is index. I can say index is equal to how many are there? uh index index index index index index index 1 2 3 five rows are there so I can assign five numbers over here and how many columns are there three columns are there so let me assign this and I'll print the data frame you can see over here every time you run it it will be different values Now in this data frame if I want to print the shape of the data frame I will say print the shape of the data print and that says five rows and three column. If I want to print all the column names because here in this case there are only three columns in real life you may have thousand columns 20 columns 100 columns 200 columns so when I want to print the columns of the data frame just like dot shape I use dot columns and that will print all the column names. Now some of you may be like daran what is this index of this? Why is it printing like this? Don't worry at all. It is panda's style of printing. And for some reason if you are not happy though it's not recommended better be happy. But if you are not happy for some reason then what you can do is you can definitely type cast it to a list but not required. This output is only what I strongly recommend. No one changes it. No one. If you want you can change it but no one changes it. So you are unnecessarily investing energy in the unnecessary steps. So you need not as simple as that. But if required it can be type casted. Next. So this is for listing all the column names. If I want to print all the indexes. Now in real life you will never print the indexes because here there are only five indexes. In real life there will be minimum thousand rows. Minimum thousand rows. So what is the use of printing it? But if you want to do that, you have the index which you see over here how it comes right. You can see how it comes. Then if you want to print all the values, see these are the values, right? This this these all. If you want to print all the values then you use the values. Now let me tell you this output that you get will be a numpy array. When the contents of the data frame are printed what you see is a numpy array. I'll print that also for you. Print the type of df do values. So you can see it's a numpy array and this is how it gets printed and there is no need to type cast it to a list but I just type cast it. It looks like this but not required. I'll better remove this code. Right now I want to talk over here that this shape or columns index and values all of these are attributes of the data frame. They are not methods. See what is a method? Did I write over here index open bracket close bracket? No, that will not work. They are attributes. Did I write over here shape open bracket close bracket? No. So these all are attributes of the data frame. These are all attributes of the data frame. And you might be curious of knowing how many attributes are there. I told you yesterday as well very less seven to eight attributes are there. Now it's like almost you studied most of them. Most of them are done. And I think one more is required which is D types that we'll study in some time and that should be end of the story. Okay. Now in real life we will never be creating the data frames from the scratch. I was very very clear on that. So how do we actually end up working on this? That is where we need to understand the things. So I want to show you uh The data set which I am downloading is available on my GitHub repository at this address. I am saving it first of all. So if you see I saved this data set. I downloaded it and this is the data set. Okay, I downloaded it. How did I download? When you click on that link, it will open this thing in the browser. Huh? Something like that will be open in the browser. You can check it out. Exactly like this. It will show in the browser. Right click on this and you have the option save as. And you save it. As simple as that. Save as a CSV file. You are done. Okay? You know this is just to tell you that hey this is what I am doing. Now I want to read this. This is a CSV file. Now there are many types of files which can be read. Right now I'm focusing on CSV file. That's going to be the most common. So the link to download the data is this. Right? This is the link to download the data. Now how do I read the CSV file? Same I say df is equal to pandas has a function read and now if you see when I type rea even in Google collab it will show you suggestions so what all files can you read you can read a cv file excel file feather file now I have never used a feather file file but if required it's not a great thing to read it. You directly have the functions. You can read a JSON file, HTML file, park file, pickle file, etc., etc., etc., etc., etc. So, I say I'm going to read a CSV file. If you write Excel over here and if you're reading a CSV file, obviously expect an error. So, read the CSV file. And then what is the name? It is always advisable that this file you save it in the same folder where your notebook is there. Else you will have to write the complete path complete path in the sense you will have to write C colon slash colon users colon SL student records CSV. Right? So you'll have to copy that part here because you are putting it in the same folder. That's not a problem. And I'm done. Now I say is the data frame red? It is red. It is red. Okay. So that's one thing. Have a close look at the code please. Okay. After this, I'll just delete this. Then if I copy this code and come down right if you are having an access to the internet then you can directly give in the GitHub URL as well to read it and it will read it seamlessly. You should have access to the internet. See both of this code here it's there reading from the local system and here it is reading over here. Right? So it can read it from the internet as simple as that. But but but if you are reading it maybe on from some Google drive or so then the processes may be different. So ideally 99% of the time you will end up reading from the very popular sources and and understand our session pandas. What is pandas for? Am I here to teach you how pandas is used for reading each and every type of file? The answer is no. Our session is about understanding how to use pandas. Reading is one time affair. Reading is one time affair. The code for reading a data can be found anywhere on the internet. If I say let me go on to chat GPD and if I say that you are a pandas expert get me the syntax and the code to read the file from the most popular sources like and more whatever you think is relevant right so maybe uh I'll take one more SQL I'll take and there you go reading is just one time a reading is not going to make you an expert in pandas what is going to make you an expert is the later part of this how do you do analysis cleaning etc etc we have a separate session on data cleaning so today we are not getting into too much of that But just remember so df read CSV and you have that basic CSV sometimes have delimiter other than comma there are also files called as tab separated value tsv so over there the delimiter is not comma the delimiter is back slash t so you can use this sometimes encoding issues come so encoding is one of the common thing usually it's not required right now when you start with the EDA machine learning at that time. Sometimes this is required. Read Excel. Excel in CSV there are no sheets one single file but in Excel there can be multiple sheets. So you have to supply the sheet name then JSON read JSON. So wherever is your JSON you can just use this function and give that name of the file over here. Right? So orient function is also there in JSON for the JSON in string format. Park file is mainly used in the data what do you call in the distributed computing because it is very fast and saved in purely columnar fashion. But anyways it cannot be directly open to see the content but it is saved in a compressed form. When you have used data CSVs are not preferred parks are preferred. So that's how you read it. straightforward for connecting to the SQL you need to use extra module import SQL light and you have to connect to your database whenever you study that at that time and then you use the standard command of SQL to get it connected for connecting or reading a HTML page you have the read HTML right I hope that gives you more clarity on that right so here you can also read the clipboard data whatever you have it on your clipboard file is again used in uh something like park only feather file I never used it it's a fast binary format but uh at least I never used it pickle file is the format in which machine learning models are saved so that can be read like this XML file and reading from the zip file. This is uh usually required when you start with the what do you call uh deep learning when the data sets are very big and you have lot of images to be read. So we usually compress it in the form of a zip file right and then after that reading from S3 Azure or blob right so this is all reading from the buckets just like supplying your GitHub URL you are doing the same thing over here now never ever in my life so far I have ever read any data from the Google drive so in in Google drive if you have some data set uh it's not that you click on share this data set and you get the public URL it will not read it from there so I have never tried it and see as I told you right if I have to ever no in none of our consultancy use cases you will ever read it from the Google drive no one puts it publicly on the drive but if you want to read it Google collab is very nice for reading the data from Google drive I'll show you that also but mainly mainly uh we never have it right in practical uh life you will never have it so but let's say you know how can I read from my Google drive to my local PS code Jupiter notebook though it's not relevant I'm it's it's like not required but you know since you are learning it for the first time you may have all these type of concerns what will happen here what will happen there manually download that's the easiest way second is there are some external packages like gown or pi drive which can be done so it's a third party package so if I am not there to help you I think it's not a big thing right it says install this package you install it and then your link is there. Let's say this is the link. So you'll be giving the file ID which comes in the URL the link over here and it will download and after downloading it is again reading reading using the CSV file. So that is how it is. If it's one time or occasional prefer the manual download option as simple as that. Now if you are having a GitHub URL right then there is absolutely no need to worry if you are having a file on the GitHub then you can directly read it on Google Collab it's all set it's all set absolutely nothing to worry because it is reading from the internet and anyways collab requires internet so that is just out of question this is sorted Second, the second thing over here is if I have to read it from the file not from GitHub then you have the option over here here or here file section. I click on this and you have the option over here to upload. You may end up asking this to me once again. So I'm telling it right now only so that I need not repeat it. You click on this Then you click on this to upload. And now I am actually uploading the file. How? By getting into our regular folder. This student records CSV file is getting uploaded and it is temporary upload. You can see the message warning message. Read it. What am I trying to say from this is if I go to the runtime because if I close it, you see if I close this file directly, if I close this file, then this runtime which is there, it will be automatically disconnected after some time. And if I disconnect it right now, if I disconnect it right now, you see it says all the local variables and files will be lost. So if I disconnect it, it's all gone. So it's temporarily uploaded. Let me reconnect now. After reconnecting, it doesn't come again. You have to upload it. Click on this and then you upload it. You should always wait for the upload to complete up. Don't try to read before the uploading is done. Specifically in case of bigger files. So that's one thing. Then how do you read it? Well, let me first copy this code. I come down. Entire code is same. If you observe carefully, entire code is same. I'll just click on this three dots and copy the path. I repeat, I click on this three dots and copy the path. control V and P is not defined. Oh, I restarted the kernel. Right. See, now it right it came. A I want to tell you that even if you don't click on this three dots and copy the path, even if you don't do that. So let's say if I am taking the same code as I took absolutely same code you can see that absolutely same code if I directly give the name of the file without giving that /content earlier I gave this complete path but the default path is /content only so even if I do it like that it works that is the third thing the default path is always content so it need not be compulsorily given Easy. And the last thing, let's say I go to see there is one more. Let's say there is an U. superers store cs file I'm reading you can directly read it the same way but what will I do is I'll first download it while you're downloading please do not click on any text it will select that text and ask you to copy you don't want to do that right so click on this empty portion and you have the option and save as. And I'm saving it. Okay. Now, what we do is I'm going to my folder. And then I'm uploading that superers store CSV file. In fact I uploaded both the files. H now I want to let's say read this file. How can I read it? Well, there are something that you have to understand over here. So where is this file? Well, it's in a lengthy path. I like to show you that this file I have saved into the folder D2 which is ideally inside our this folder which is ideally inside our this folder which is ideally inside simply learn folder. So overall friends my drive path is very complicated. In my drive, in my drive I have created a temporary delete folder. Inside that there is a simply learn folder. Inside that there is a folder. Inside that this is our folder and inside that this is our folder. And inside that I have the file. Why am I telling you that is now I want to read that file. One way is as I told you I can directly read it. I can say ss is equal to pd readad csv and you have the file read directly from GitHub right directly from GitHub you can see that okay it is a big file 10,000 rows and 31 columns in fact all the columns are not even displayed neither are all the rows displayed but I want to now read it and let's say the size of the file is big. So you know what can I do? I have this option mount the drive. So whenever your file is let's say 800 MB 900 MB then uploading this file would be taking a lot of time. If you already have that file in one of the drive folder if you already have that file then it's better that you click on this. It is saying mount drive. When I click some code will appear in the notebook. Connect to drive. I say yes. It says mounting the drive. No code has appeared. It used to appear earlier but it no code appeared. Earlier a code used to appear and I had to run it. They automated it. That's good. And then I have my Google Drive mounted over here. I click on this drop-down. I click on my drive. Then I look for temporary delete folder. Then I look for the simply learn folder. alphabetically towards the end look at this IITK because I gave a very complicated path then if you give a simple path then you may don't you may not have to do so many things and inside that I see this things I want to read which data set supertore click on this three dots and copy the path that's it so first you have to do the mounting And after mounting you go to that path and copy the path. Then after that I can actually say SS1 is equal to PD dot read CSV and this path which it will be pasted by that path very lengthy path in my case but in your case it may not be such a lengthy path and I read it. So every time you will have to mount your drive for reading the data. Every time you will have to mount your drive and your path will be different that mine path. So these are the different ways you can think of reading the things in collab. Right? So sometimes uh you will also see the files are zipped. Right? So files are zipped. So these are you know another scenario where you have to unzip the file and read it. But yeah essentially these are the ways you do it reading of the files and everything is done and this is my data frame so far. So now we understand the various ways of reading the file and all these things are done. Now if this is the data frame, I'll read one more data frame which is the super store and let me get this super store in the uh Excel file mode. So I'm downloading it so that I can show it to you here only. Superstro.xls SS is equal to PD dot read excel. The name of the file is this. [clears throat] Excuse me. And I just say ss and as expected pandas require 3.1 or newer version of pi xl to read this file sheet name pi excel I don't want to go open py excel uh it can be read using the panda pandas as well. Maybe I have to upgrade my pandas version. So I say pip install pandas upgrade printing the version. It's this one. So this is the latest one specifically. So, but it can be read in Excel as well. Maybe the version is older. That's the reason. But that's not a problem. So, you all can see that this works. Maybe my version of pandas is little old. Uh let me see what version it supports. Ah, it's actually dancing up and down. I don't know why. I click here. Huh? 2.2.2. But anyways, it was giving some trouble but you have it over here, right? So, this is how you can read an Excel file and there are many other thirdparty libraries. You can use open py and there are many many more libraries other than open pixel which can be used for reading the excel file. But we are here to only understand pandas at runtime. Sorry for saving it. Runtime disconnect. So the code successfully works. That's completely fine today if it didn't work on my system. But uh I'm sure everyone followed how you can read a CSV file as well. I hope it's clear to everyone. Let's go ahead. And pandas is like the heart. very very important. So if you follow it that's awesome. So now we have our data frame here. Now what is this data frame? See this data frame is about a student. There is a student called as Henry. His overall grade is A. He is obedient. He scored 90 marks in the research course subject, 85 marks in the project score subject. Based all that will the university recommend him some kind of grant or maybe some admission fee discount scholarship etc. Answer is yes. David on the other hand overall grade F not obedient bad marks 10 out of 100 17 out of 100 recommendation no in this way I have a toy data set. You know why am I considering this toy data set? The only reason is I also have this superstore data set right. So I do have this super store data set from where I can read this. The reason I have chosen going ahead with this toy data set is because it will help me understand. For example, for example, if I ask you a question, can you tell me in this data set which is having 9,994 rows, what all are the unique values in the category column? Can anyone tell me by looking at this? What all are the unique values? No, because you don't see all the 10,000 values. Right? On the other hand, I tell you that can you tell me what all are the unique values in the recommend column? Can you say in this case because this data set only consist of eight rows? It only consist of eight rows. Yes. So the only intention behind choosing a small data set while understanding the library is we are able to crossch check our results. Whatever code I write, it will work on this data frame or it will work on a data frame of any number of rows. The reason for choosing a small data frame is understanding. Okay, don't forget that the only reason for choosing a small data frame is understanding. Else absolutely there is no reason. So here so we have all of these things in place and I was saying we have this data frame as well. Now in this data frame if I want to print the shape of the data frame I will say tf dot shape done eight row six columns. In the same way, if I want to print all the columns of the data frame, I will say df dot columns. That is what I will write in the same way as I told you. If I want to print the values of the data frame, I will say df dot values which is this numpy array. If I say I want to print the index of the data frame, I will say df.t index. We have already seen that right? It goes from 0 to 8. This time it's not a b cde e. If you recall, it's going from 0 to 8. So it is saying that starting from zero going to 8 minus one with a step size of one. Okay. If they are ABCD or anything else then it will be displayed properly one by one. Okay. Now the thing is if I have to select one single column from the data frame [clears throat] then I would like to propose you two ways. The first one is using the column name as an attribute where the syntax is df dot column name. Let's say let's say I want to select the recommend column. I want to select the recommend column. So I will write df dot recommend. I'll write df.recommend. And you see the recommend column is selected. very very easy selecting a column of the data frame because to do some operation you need to select that so DF dotrecommend is for selecting one column of the data frame and overall overall if I look into this the type of DF is a data frame okay and data frame is two-dimensional in nature you know what what will be the type of this it will be a series type of this will be a series. Listen to me very carefully. This entire thing is a data frame. Every column in the data frame is a series. So that's one series. Second, third, fourth, fifth, sixth. So this data frame is a collection of six series. data frame is a collection of 6 series. Okay. Now this is the first way and I would like to say that it is not recommended. Now you'll be like it's so easy why is it not recommended? No problem. I will like to go on to the second way. The second way is using the column name as the key. Now that may look a little bit complicated but this is the best way that's like this DF square bracket and in double quotes you write the column name. So you want to select the recommend column you will write it like this and not like this and this is what is recommended output will be same. You can see the output is the same. Even if I look into the type of this, it will be a series only. But why is this recommended? The reason this approach of selecting is recommended as compared to the other approaches is because see here in this data frame if the name of the column to be selected is not recommend but if it is research score then df dot research score I get this but imagine the name of the resource score column is not like this but it is research space score then it will give an error I can show you that I know that df doc columns gives me all the column names I say df doc columns is equal to paste it and I say Please make this research score as research space score overall grade like this for better column readability and see everything has changed. So this is kind of like changing the column names and I was able to put in spaces. I can replace it by anything but I just inserted spaces. Now in this case if I say df square bracket resource code because it is contained in the single quotes it works right so it can be resource score can be in double quotes or resource score can be in single course that's not a problem but if I say tf dot resource score it can fail if I say df.res resource school it can fail maybe you know I'll just comment it out that's a clear case of syntax error therefore I say it's not recommended maybe if you write something like this it was expected to work but that also doesn't work that is the reason I said that the first way which is square brackets double quotes is the ideal way to select I use any one of these ways. I mean I can use this one as well. I mean to say I can use this one as well or I can use this one as well. I recommend you also but just be aware that when there is a space the first one doesn't work. Okay. So that is for selecting one single column from the data frame. Okay. What if I want to select more than one column from the data fit? So what all are the column names? Well, uh the names of the columns were obtained like this, right? So let's say uh I want to select the columns of name, overall grade and I want to select let's say recommend column. Okay, these are the three columns which I aim to select. So these are the names of the columns. So what will I do? I will select a I'll create a list columns to select. Huh? I have created a list which is these are the columns I wish to select. And then in the data frame I will put the columns to select. And there you have it. In the data frame I put columns to select. And there you have it. Look into this code. And because this columns to select is taken from here. Then now I'm thinking that what if I actually cut this and just remove this and here only whatever I did cut I paste it. Essentially I did the same thing. These two are the same. I just replace line number two's content directly inside this. If you see directly I replaced it inside this. So what do we understand? For selecting one column, you are going to use single square bracket. And for selecting anything more than one column, you will be using two square brackets. This is what we understand, right? One column or more than one column. And that is how you select it. After this next we say over here that's my data frame. Now if the data frame is SS it's consisting of these many rows and these many columns. Right now here obviously it's taking a lot of space on my screen. So what I want to say is if I want to print the first five rows of the data frame I use the head function of the data frame. So ss dot head and you can see it only prints the first five rows. And inside this if you supply number like two it will read only the first two rows. Inside this if you supply number like seven it will read the first seven rows. Can you supply 8,000? Will it print all the 8,000 rows? No. Max to max number which I have tried from hit and trial is 60 6 but don't do that so you know see this data frame is basically for understanding how the data looks like it's like how are you recognized at the airport you show your passport size photo this is me that is the only purpose of this head function it just shows how the data frame looks like and if you have to look at the last five rows of the data frame then it is the tail function last five rows and in tail also if you give a number like three it will show you the last three rows then this is what we understand now with respect to the SS do head only if you see carefully I'm not able to see all the columns of the data frame because there are total 31 columns now this comes comes with a very simple uh code to be added. So if you want to display all the columns of the data frame then you will have to set the pandas display options and I'll say over here TD dot set option display dot max column, none. None means maximum columns display. If you write over here seven, it will only display maximum seven column. But you will always write none only. And now if I say SS do head of three, you see all the column names. You see a scroll bar as well. So category, city, country, customer ID, customer name and so on and so forth. This is how the content of the data frame looks like. This is how the content of the data frame looks like. Okay. Now after this getting back to my data frame DF. Now uh we understood how to select one column. If I want to look into the data types of all the columns then I have the attribute called as D types. If I just say DF dot dtypes every columns data type is printed. So research score and project score are integer depending on the data it can be float also. If it is 90 something then all other columns are string but pandas calls string as either an object or category. So don't be worried about it. It is nothing but of the string data type or the categorical data type. Okay. So that's one thing which I wanted to show you over here. So this is all done. Then after that if you want to print the information about the data frame you have the info function. And this is interesting because if you look carefully the output of D types output of D types is again printed here. So what all you see over here it says that hey it is a pala score data frame. Second it says there are eight rows going from 0 to 7. Third it says there are six columns. Next it says these are the names of the columns, data types of the columns. In every column how many nonnull values are there? Like in name column it says eight non-null values are there because in our data frame df we know that there is nothing missing. Are there any non-null values? No. If there are any non-null values they will be saved as nan not a number. But we can see the entire data frame. So you don't see any non-null values. It says that there are two columns of integer data type in this data frame and four of the object data type. The total memory usage is 516 bytes. And if I try the same thing with my super store data frame. See total 9994 are there. But in the order ID it's clear that there are missing values. That's why it's saying 800. Right? In the same way going forward you can look for other things as well. So we understand clearly the data types are there. The size obtained is 2.4 MB plus float data types two and this and this is what we understand from this. So this is over and then df info this this this yes if I want to find out what all are the unique values in the recommend column I can say I want to find all the unique values in the let's say recommend column. So I'll use the unique function and df dot recommend dot unique. That's it. So I know it's only yes and no. And it says that and the output is a numpy array by the way. Oh, you don't be worried about it. And if I want to find the count of unique values in the recommend column, I use n unique function. So there are only two unique values, right? Yes and no. So the count is two. Yes. No. Unique values. You get that? So in the superstore data set, I remember there was a category column. What all were the unique categories over there? office supplies, furniture and technology. And if I use the n unique over there because it was a big data set, there are three unique categories. And if I want to print the count of unique values in each column of the data frame or let's say let's say you know I I'll be very simple here this recommend column is giving me when I say df do.recommend do unique what is it giving me? It is telling me what all are the unique values yes and no. Instead of this unique if I say value counts so value counts will be counting the unique values in the recommend column. Have a look at the output and that's what will be returned. That's what will be returned. Whatever you said it returned that. Perfect. Five and three. That's what it return. And there is there is a parameter inside this recommend value count which is called as normalize. And the default value of normalize is false. I want to tell you that normalize equal to false is the default value. So if I run it, you don't see any change. You do not see any change with respect to these two codes. 53 is the output here and here also it's 53 but if I write normalize as true then it gives me the percentage you know what is it giving I'll tell you how many times no is there five five times no is there out of total eight records so five upon 8 is 62.5 percentage wise 62 32.5% of the times that column contains no and 37.5% it contains yes. So this is what it is returning me. And if you want it truly in the percentage scale of 1 to 100 then you can definitely take the same code and multiply it by 100 instead of 625 you get 62.5 and so on. Very easy and very convenient. very easy and very convenient, right? So this can be helpful in this category column as well. So in this category if I apply the value counts out of 9,994 records I know that most of them are office supplies and the least are from the category technology. So it's very easy to identify at a scale. Right? Now in this data frame if I want to find the missing values I have the function is null. So it will tell me where all missing values are there. Wherever you see a true missing value is there. You see anywhere true? I don't see. I don't see even I don't see. But if I do the same thing on the super store, I will go mad because there are 10,000 rows and 31 columns. I don't have time to keep on looking for true true this way. I don't have that much time. So what will I do? I will apply the sum function ahead of this. So sum function will be telling how many times in this category column true is there. How many times in this category city column true is there? How many times in this region column true is there? Every column total that's nice. So order ID column has 91 94 missing values return also has this. Now it's very difficult for me to keep on searching. So what am I planning to do is this this entire thing can I sort it in the descending order and I can do that using the function called a sort values whose ascending is by default true. So I say descending. So what happens is it is going to sort this earlier printed value using the sort values function. Ascending is by default true. So I'm writing descending which is nothing but ascending is equal to false. And now because I know that I got the first zero over here and because this entire thing is in the descending order I need not even look below because everything is already in the descending order and that is what is the idea over here. Hence the thing. See now over here I want to say some important thing. Listen to this very carefully. Now here in this case see these are 0 1 2 3 4 and these are the ones internally it also maintains numbers and so these are the internally maintained numbers for every row and every column. If you want to refer to those you have to refer to iOS which means the ones which are shown over here and here you will have to go for lock. What I mean by that is let's keep our target very simple. So my target is to get everything from Henry tomorrow. So this is what is my target. How can I do it? I say DF dot df do. I L O C I have to give the rows to be selected and the columns to be selected because I'm using the implicit location. Okay, eyilock means implicit location. So which rows I want to be selected? Well, the rows which I want to be selected are row number zero to row number four. I'm referring to the ones which are here. All right, I want row number 0 to 4 which means my starting index will be zero and my ending index will be pi which means it will go till 5 - 1 and which all columns I want to select column and I want to select 0 1 and two so I want to select column 0 to two which means my starting index would be zero and ending index would be three. That is how 0 to two will be selected and here 0 2 5 will be selected which will take up to four. Let me try it out doilock row number 0 to 5 - 1 column number 0 to 3 - 1. It should give me Henry till Marvin Y till N. Does it? It does. Did you all follow the eyelock which are nothing but the implicitly given numbers which are in the yellow ink? which are in the yellow ink. Yes. In this way you can select anything. You can select anything. One operation is sufficient. You can select literally anything but it has to be consecutive. You cannot say that can I select one row and then third row and then fifth row. No, that all things are not. That is also possible but right now we are not getting into it. Okay, this is clear. Now I'll go ahead once again print my data frame. Huh? I say DF dot lock rows and columns. Which rows I want to select? Now refer the white one 0 to 4. So rows I want to select 0 to 4. And the columns which I want to select are what? Right from name to obedient. So start would be name and end index would be obedient. Let me explain you it has fed the same thing. First of all something interesting the interesting part is in block in lock for which I am actually using the pinking in lock the start index had to be zero and the end index had to be four and while giving that I actually gave the same it I didn't give five earlier I gave five and Here the start must be name and the end must be obedient. Exactly. That is what I gave. I didn't give obedient + one or something. That is the different right. The difference is in eyelock the start index is start index but end index is considered ideally as end index minus one. Whereas in block start index is start index and end index is also end index. Why? Obviously you cannot say obedient + one or resarch score minus one that all things. How can you do resource score minus one? That's not allowed. So in lock start index is start index end index is end index. This is one of the very important observation. This is one of the very very very important observation. have a close understanding of this concept. What is start index and what is end index in each of these scenarios. So now I got this I am going ahead. Now here in this data frame not data frame you know I'll go to the population you remember population was a series that we created at the start here this this one we created a dictionary and from the dictionary we created a series I'm talking about this right this now inside this population also these are the indexes So here I can say population dot iO. So if I have to select California, California ideally is at index zero. You see I'm getting the data for California. Right? And if I got to do the sum same thing and I want to write over here California then I will have to use lock gets the same information right. So this is internally 0 1 2 3 4 5. This is what it is internally. Okay. In the same way, if I want to select the data from California to Florida. So, California is at index zero. Texas is at index one. Florida is at index 2. So that means I will have to write population dot eyelock and California is at index zero and Florida is at index 2. So that will select the entire highlighted portion. So this this if I want this as the output then I will write population dot it has to be three absolutely correct. Yes. and and and if this is the code for iOput lock is about using the externally given indexes which is California, Texas and Florida. Now we get into some interesting operations. So what I say over here is so we have a list of students. We have a list of students. Now what am I planning is to create their report card. First thing I am planning that I would like to find out the total of this and create a new column over here which is called as total in the sense 90 + 85 should come here. 85 + 51 should come here. 10 + 17 should come here. 75 + 17 should come here. 20 + 30 should come here. So on and so forth. How do I do it? Very simple. Research score and project score has to be total. So I write research score like this and I add it with the project score. Does that give the total? Yes. Did you notice over here? The beautiful part of pandas is no for loop is required for doing the operations across every row row. for loop it's not required it can still manage it there is a for loop which is running internally because of which it is actually to are you all following it internally for loop so it's all basic code is ready you just have to do the addition and your job is done are you clear so far how did this addition come without a for loop very very important to know internally there is a for loop When I learned pandas for the very first time like an idiot I used to actually write a for loop to find the total because I was very much habituated to using C language C++ language. It took me almost a month to understand that all of these are advanced level coding and I need not think of writing the things from the scratch. So I had a very bad experience when I learned pandas on for the very first time. It took me almost 3 months to be very comfortable with it. I mean just playing around with everything. It took me 3 months. So but one thing is remaining. This is okay. But I want this to be saved in a new column. So I say what is the new column that I want to create? Total. So d of total is equal to this. I'm done. It will automatically create a new column. Best. Best. That is the best part of it. You don't have to worry a lot about it. it will automatically create a new column. How convenient and easy it is. You need not even write a code to create a new column. Huh? If you say that I want to create a new column and insert it in between these two, then the things are different. Huh? Then the things are different. There is a function which is called as insert for inserting it at a particular index. But by default it is automatically inserted at the end. So let's say you want to create I'm just doing a dummy column. Dummy dummy dummy. Okay. So I want to insert a column. Let's say internally we know right this is 0 1 2 3 4 5 6. Let's say in between name and overall grade I want to create a column called as dummy which will just have one single value one one one one just like that just to show you just to show you that I want to insert a column dummy in between this and I don't know how to do it huh let's say I don't know how to do it I am making you ready to solve any problem by If I am not there, how do you survive? go to Google and search for insert pandas or let's say you don't know that also right so I say insert column in a data frame right you don't know what to google you don't know the name of the function nothing insert column in a data frame panda says I have a method so this gives me some idea that okay I can do it this way geek forge geek is also giving me some help. I'm opening it. W3 school is also giving me some help. Let me look into the first help. Insert insert a column at a Okay, I got it. Insert a column at a specified location. And what it says first you tell me at what index you want to insert. So that's one thing. Second, it says tell me the name of the column that you want to insert. Third, tell me the value of the column that you want to insert. And do you want to allow the duplicates? Uh you have over here true or false that can be given. Okay, I understood it. In fact, there is one example also given over here. Okay, that's one thing. Let's say I'm just timing putting this Then this gig forge geek is also there to help us out. So it says that if you have a data frame like this and you want to add a column with static value to the pandas data frame, what have they added? Let me see what was there. Age, name, and address. Age, name, address. Okay, they added it at the end. Okay, that's okay. Then insert. See here. That's good. This is a good resource. W3 always is awesome. It says they say you have a data frame. I tried it. This is a data frame. Name, age, and qualified. And okay, name and age are only there. Qualified is not there. Let us print the data frame again. And let's run it. You can see name and qualified were there. And now it is name qualified age. It says you have a data frame like this. I say I want to insert at index location one the new column called as age. Okay. So this is zero column. This is the first column at first column. That means this column will be shifted and the values will be 50 40 30 and that is how you insert right. You can also insert it on the basis of some condition but I think that we can take it later. And now that we have it I don't wish to execute it because we already have it. So I will better comment out this code. You can run it at any point of time. The reason I'm commenting it out because the name of the data frame is DF. If I execute this code, then my original data frame is also called as DF. That would be lost. So I will have to change the variable name. But anyways, I did show you the output. So I don't think so I need to execute it. You should be able to automatically figure this out. Okay. So that's my data frame so far. Over here I want to have one more column over here which will be called as percentage. So how is the percentage computed for Henry so on and so forth computed for everyone? So the way I do it is DF of percentage is equal to DF of total divide by 200 into 100 exactly like this because total value divide by 200 into 100. You need not write 175 136. You just give the column name and it is good to do its job. You can see this. So what I want to do is I want to find the topper in the class. So I will consider sorting this data frame in the descending order of percentage. Right? A very realistic operation to sort it in the descending order. So I say that I want to sort the data frame in the descending order of percentage column. So the function that I will be using will be the sort values and you have to sort it by which column it has a parameter by. So I will supply that I want to sort it by the percentage column and ascending is equal to true is by default. If I say ascending is equal to false it will be doing it in the descending order. So this is how I do it. Okay. Now pay attention. The sorting happens but does not happen also. I'll tell you what I mean. df dots sort values by percentage column ascending is equal to false. And you see that it's all in the ascending order. A descending order. All perfect. Let me print the data frame once again for you. Is it really sorted? Answer is no. It's sorted but then when I printed it again it is not sorted. Okay. So I was saying we actually sorted the things and after sorting we saw that the sorting result is not saved because when I printed it I don't see this right and therefore we see that the changes are not getting saved. So what will I do? Well, I'll tell you what went wrong here. When you said sort it, it actually sorts. It actually sorts in the descending order, but it is not saving it. So you will have to write df is equal to df dots sort. So what will happen is after sorting whatever is this result, it will be saving it into df. Okay, that's one way of doing. That is one way of doing. After sorting you can save it. Data scientist use this second way in place equal to true. Both of these way are equally correct. It is not that one way is efficient over other. I'll tell you what is happening in this case. You sort it on the left hand side and whatever is the result of the sorting you ask it to save it to the data frame. And in the second way or both of them are same equally efficient you're sorting it and I say that hey whatever sorting you are doing do it in the same data frame do it in the same data frame so this is a shortcut of writing you know after writing this code you need not come back and write df equal to so this is what people prefer but both of these ways are equally same it's not that one is efficient over other as simple as that and therefore now when Okay, execute it. Uh oh, what happened? DF dot sort sort additional space here. And now when I print it, I use the second syntax with in place equal to true. In placement sorted in the same data frame. And you can see over here results are here. It is just a way of quickly sorting and ensuring that the changes are saved in the data frame. You can use any one of these two approaches. Both of them are same. Both of them are same, right? Yes, both of them are same. I would recommend you to use the approach on line number four, but both of them are equally efficient because they are doing essentially the same thing, right? Yes. Perfect. Now, how do I know there is in parameter? Well, you can Google about it. Here sort values. You see over here there are so many parameters in place equal to false by default. And what does that mean? If true it will perform the operation in place. Okay. Then one interesting thing the index numbers have changed. Earlier it was going from 0 1 2 3 but now they have changed and that's absolutely phenomenal observation. So that's absolutely correct. So what I want to do is I don't like this. So I will like to reset the indexes. So how do I reset the indexes? I say df is equal to okay first let's do it shortcut df dot reset index that's it let's see whether it resets the index dataf frame dot reset index it does see the indexes which were this so this is a perfect function right this column which was here actually got inserted as a new column here. There should be the complaint. Yes, one extra column came. Yes, that's a good observation. Okay, but but before I do that, uh first of all, I will have to save the operation. So, I will say DF is equal to DF dot reset index or DF do.et reset index and I say in place equal to true any one of this approach this time I'm using the above one and then I say show me the df okay now there is one problem that we have one additional column inserted now I want to talk about dropping a column so for dropping a column which will be very useful while you are doing data cleaning where let's say there are many columns and you don't need some of the column or So you want to drop the less important columns. I want to drop the column. So how do I drop a particular column? So the way one can drop this column is use the drop function. So drop function is used to drop the column. So it is used to drop either a row or a column. Now how can the same function be used to drop a row or drop a column? Well, because it has a access parameter. So it has a axis parameter where if you are supplying axis is equal to 1 then it is rows and if it is zero it is equal to columns. We don't need all of these parameters. So I don't want to confuse you right now. Okay. Okay. So, whichever columns I want to drop access now here in this labels is which row or column you want to drop axis is this. Let us talk about it. Let us assume assume I'm not actually finalizing this operation but let's assume that I want to drop row zero and row five. Why? Just to show you I'm not permanently say doing this operation but I want to drop this two. So what will I do? df dot drop and which all rows to be dropped? Row number zero and row number five. row number zero and row number five because I have to drop the rows I will write axis is equal to zero and I will not write in place equal to true else the operation will get saved I just want to show you so rows 0 and five are they getting dropped this zero and five don't refer the index column the act original indexes now you can see that zero and five are no more there so in this way you can drop in the rows okay in the same Okay, operation is not saved. We all know that it's just a temporary operation. I want to drop a column. So I say df dot drop. And what I want to drop, you write down the name of the column. The name of the column to be dropped is index. So I'll write index access equal to one in place equal to true. See now if you want to drop more than one column, no problem. This is a list, right? Keep on supplying as many name of columns you want to drop but that column should be there. If I say over here that could you drop for me the column index one there is no index one column it will give an error key error that key doesn't exist but index column is there it stopped. Okay, let's say I'm adding a dummy column df of dummy column which is just some constant value 100. Just a constant value 100. You see a dummy column is added. I want to drop that. Now you know what is the uh code to drop it. But I want to show you a different way. TF dot drop and the column to be dropped is dummy. So if you see over here I do have a columns parameter also. If I don't want to go ahead with this style with the access you can just say columns is equal to this. And now what will happen is it will still work without access because you clearly said no that I want to drop the columns by this name. So when you say you want to drop the column there is no question of access. So access becomes optional in this case. So we saw two things how conveniently a dummy column can be added and how it can be dropped. So I understand Henry is the student who has got the highest percentage. Okay, that is something which is very clear from this. Let's take it away. Uh how to drop the row in the second method? You have rows equal to also as one of the parameter rows equal to. So let us consider doing that. So let's say I want to drop the row three. John's row has to be dropped. So drop row number three. So you can put it this way. I'm not saving it. Sorry, I don't want to save it. I just am trying to show you. That's how you can do it. So John is out. we have this and percentage column is there. Now percentage is also sorted in the descending order. Now what I want to do is I want to round off the percentage column. So for rounding of the percentage column I have two ways. So the first is I can say when I say round off it will round off to two decimal places. This will round off to three decimal places. And if I don't give any number it will round off to zero decimal places. that I will say I can also use the round function of numpy where I say print. Now you can see round and np.round both are doing the same thing. Which one will you use in a long run? Which one will you use the round or np.round? And why? Why performance faster? That's correct. So we will use the numpy. Now in this data frame it does not matter because we have only eight records. But in a long run you will use numpy. Right? Now you understand the true idea behind numpy. That's good. So friends I have my data frame over here and I want to round off the percentage. Now why do I want to round up? Because I want to show you that operation. Else ideally one never rounds off the percentage. That's quite logical. But let's say I want to show you that operation because I am trying to give you maximum exposure to pandas. So I say BF or I will say round the percentage column to zero decimals. So DF of percentage is equal to np dot round df of percentage. You need not even write this zero. That is still okay. And it is done. I'm only working on the percentage column. Rounding it off. And it is done. Uh after rounding it off, uh I want to check the data type of every column or specifically the percentage column. I want to check the data type of that column. What should I do? So yes, there are two ways. D type the types are not DT type D types in numpa it is DT type here it is DT types so I can say DF dot D types that's one way and that's perfect over here I would say and it says percentages of the float or I can also say DF.info info which will give this. Okay. So but the question is when you look at the data frame is the percentage column now even expected to be in float. Do you think we should keep it float because anyways there is nothing after dot right. Yes. So I want to or I wish to now convert the percentage column to integer. And for that you have the as type function. TF of percentage dot as type to integer. You can also write in double quotes integer. You can also write in double quotes in 64. But why are you complicating the code? This works. and percentage is converted to integer. Now what I want to do is I want to add a column which is called as res pass or fail. I want to add a column pass or fail. So I am creating a userdefined function called as calculate result which will be taking in the percentage column and if the percentage is greater than equal to 40 it will return pass else it will return fail. Very basic userdefined function which is going to be helpful for adding a new column called as pass or fail. And I want to apply this function and that is how I do it. I say on the percentage column I want to apply the calculate result function. So see what happens you know let me explain you what happens. First the number 88 right df of percentage.apply calculate result. So number 88 is being passed here. It checks is 88 greater than equal to 40. Yes it returns pass. And that pass is saved into new column called as result. Then it sends 86 that will again evaluate and return pass. So on and so forth. It happens till 14. When 14 is sent 14 greater than 14 no return fail and that is saved here. That is how this column is that is how this column is populated. And if you note over here while giving a call to calculate result I did not call a function like this. Usually you call a function this way, right? This is how you call it. But you uh if you give that that's going to be an error. This is the right way you can call it. Apply is a advanced level function. It says you need not give round brackets. In fact, if you give it, I will give you an error. If you give it, I will give you an error. So you should not give that and this is the reason we studied the lambda functions. This is the reason we studied the lambda functions where see if you remember lambda function what was the lambda function syntax we had lambda arguments and expression so I can achieve the same task using the lambda function I am saying df of result lambda because that is already computed C lambda percentage that's the argument if the percentage is greater than equal to 40 return pass else fail so instead of writing a different function lambda function can be integrated because this is one time used logic I'm not going to require again and again and that are or this is the ideal place where a lambda function is helpful and then I want to drop this column which is quite obvious that I need not talk about. So I have the result of the student populated so far. Let's go through what all have we done in the previous class and have a very quick glimpse of it. Pandas consists of two data structures. One is the series and another is a data frame. That's one thing that we understood. And then this is how a series was there. A data frame consists of series. That was another thing that we saw. Then you this is the syntax for creating a series. PD dots series. you give in the data and you give in the index. So that's what we have in over here. We also saw how do we do slicing and indexing in series exactly same like list in series we can give indexes of our choice that's another thing that we have seen and that successfully works right that's how it comes then series also supports broadcasting so if you have one number and multiple things number is broadcasted Then we understood a data frame right the syntax for creating a data frame you give data index and columns this population series that we had. See always remember a series will be represented like this. It will always be like this because going forward down the line you may experience it. So whenever at the sorry not even this I would say whenever at the bottom you see d type return with the values above it always assume this to be a series and a data frame looks like this. Then after that when you have a data frame like this as you can see these are the rows and these are the columns. Then you can give the index and the columns as well while creating a data frame. Then I can read a data frame from the GitHub directly by pasting the link as it is over here. Or you got to supply the complete path. Uh we saw right you can read a CSV file, Excel file. All this was RGBT generated content I quickly pasted over here for your reference. JSON file, park file, park is used in mainly the Apache formats. Apache formats in the sense when you are working on a distributed system park file save the same CSV file in a compressed form. So that is the biggest advantage of the park files and then SQL databases you have to import a module and get it done. You can read an HTML file. Then lipboard data if any table data is copied. ORC again it is used in the sense of compressed format only. Feather data set mainly useful for the pipeline stages. Pickle file for machine learning. XML we know Google sheets also can be read by giving this you know this format equal to CSV. These things are important. Zip file can also be read using the read CSV zip and then you have to unzip it then Amazon S3 right and better option for Google drive is to download and consume it right that's one thing I told you or Gdown is the package which can be helpful after reading this data frame we also read a supertore data frame you know we may end up using them little till today as well because we are left out with little bit of pandas. So that will take another 1 hour I guess 1 hour 1 hour 15 minutes for completing pandas and then we get started with the matt plot lib shape will tell me rows and columns columns will only print the names of the columns. Now values the moment I say df dot values the output that you see over here is a numpy array. Always remember the output that you see is a numpy array. To select a single column you can use dot or you can use square bracket double quotes of which the recommended syntax is this because if there are any spaces then this syntax is better. Then you have to change the names of the columns you can write dfc columns but you can also change the name of the columns using the rename function. Right? You have a rename function which can be used for renaming the names of the columns. So here rename panda's data frame column you see over here rename function it needs a mapper. Have they given any example? Yes. So if you see over here this data frame is been created. So column A column B right? I hope you recolct this is column A having no oh sorry. So this is column A having the values 1 2 3 column B having 4 5 6. So I say rename columns is equal to A will be renamed as small A and B will be renamed as small C. So that's how it come. Initially the column names are capital A and capital B like this. But I renamed it to small A and small C. for selecting multiple columns. I said it's always good to use two square brackets instead of one. Everyone knows that very well. Then if you want to see the top five rows, use the head function. Bottom five rows, use the tail function. And head can also be supplied with a number that will display only that many number of rows. The maximum number that we can give over here is what? What is the biggest value that I can give in the head? head can display top five, top six, top seven what is the biggest number that one can give in the head and don't trust me try it once you never know in the updated version of pandas they this has changed but I think they won't change it usually they don't change all the small small things but you try it once on the super store data frame you can definitely try set option set option is for displaying all the columns because earlier we see dot dot dot in between the column names So set option display dot max column will display all the column because the second parameter is none. If I supply 100 over here, it will only display the 100 columns out of let's say 500 columns, let's say. So you can choose that number over here. Dypes, we know all the data types are listed. Info is the information of the data frame. These are very common functions. you need to use it every time in any project then you have to find out the unique values in the recommend column. So that's done using unique function. If you want to find out how many students were recommended and how many are not that is how many yes and how many no like here we can clearly see 1 2 3 yes and five nos. If you want to have that and that too in the descending order of descending order then value counts as the function whereas unique only prints the unique values after that the value counts can also be given normalized which is by default false if I keep it as true it gives the things in the form of percentage multiply by 100 will get it on a scale of 100 so 625 becomes 60 to.5 n unique function you know see these all functions you may be like these are all so stupid silly functions. Why do we even need it? Imagine a data set which has 50,000 rows or five lakh rows. How will you know how many unique values are there? Because because going forward, you know, you'll be looking at a technique like get these. I'll tell you what it is. Machines are machine says dian we do not understand English. machine says dan we do not understand English we only understand numbers we only understand numbers so what they do is if you give it any kind of column which has categorical variables like for example this or I'll take one more simple example this. Now if I ask you how many unique values are there in the six column, what will you answer? How many unique values are there in the sex column? Two. Very good. If this data set had five lakh rows, would you have been sure about this number two? If this data set had five lakh rows, would you have ensured that there is only male and female? No. Right? This is still okay. This is still okay. But you know what happens when you start with machine learning? Machines say that dan I do not understand English. So please convert it to equivalent numbers. What I mean by that? Let me explain you. So what the get dummies function of pandas does we don't have to study it today it will come in the machine learning part but what does the get dummies function of the pandas does over here is it will find out the unique values in this so the unique values are found out to be male and female right so it will create two columns sex male sex female and then it looks into the first one is male. So in place of male it writes one female zero. Second is female. So in place of female it writes one that's zero. Third is female. In place of female it writes one that's zero. Fourth is male it writes one over here and zero. Now this means this means that one column is converted into two columns. If there are five unique values then one column will be converted into how many columns? Five columns. Now you will be like okay fine five values are still good. I can go ahead with it. If in a column let's say instead of sex instead of sex the column is the products and that company manufactures 50 different products. So this products column has five lakh values and there are 50 unique values. Now in that case your data set has one column called as product and there are another 50 such columns which are different not products could be sales etc etc but before machine learning one column would be converted into total 50 different columns. One column would be converted into 50 different columns and that is not a good idea. That is not a good idea. over there. These functions in machine learning are very very helpful because you can use the n unique function to find out how many unique values are there in one column and then if the number of unique values are way more higher that means if you transform it using the get dummies and it is going to become one column becomes 50 columns then it's not a good idea in that case what is recommended is instead of dummy ification you go for label encoding. So what does label encoding do? Label encoding is going to see that okay what all are the unique colors? Red, green, blue, red, red, green, blue are the unique colors. It assigns one number for each. Red is zero, green is one, blue is two. What in the order it receives it assigns that number and then entire column is replaced with the mapping as red as zero, green as one, blue as two, red as zero. So wherever red comes in this column that will be converted as zero. Wherever green comes one, blue comes two. So one column is still one column. Now in which case to use which technique that's a different story but I'm just trying to make you understand the significance of n unique. So n unique is number of unique values. You can understand the practical significance of it. That's the only idea behind it. Okay. We will be coming back to this technique later on when we start into the feature engineering session. Right now it was just to make you understand that all of these functions exist for some reason. Then unique will print out the unique values and unique will print 1 2 3 there are three unique values. Obviously we're looking in recommend. So there are yes and no two unique values in nun of the category 1 2 3 unique values. Then isel dot sum is for finding out the missing value the count of the missing values or the sum of the missing values in every column. It's always good to have this in sorted order. So we understood the sort values functions which will be sorting it to the descending order. Then this eyilock and lock were very helpful. Eyelock is implicit indexing. LO is explicit indexing. So is used for implicit indexing. So it will be accessing implicit index 0 and five that is rows from 0 to four columns from 0 to two and log will be actually going from row 0 to row four only and column name to obedient. We had seen that. So let me not get into the teaching part of it once again. We are just getting into the quick revision. Then we understood that we quickly totaled up these two columns and created this column like this. Then coming over here total upon 200 into 100 gave us this new column called as percentage. Then we sorted the percentage column using the sort values function in the descending order but this does not get saved. So you need to write preserve the changes. This is also one of the way but I would recommend this. That's why I kept it uncommented because you know when you're typing the code you type it left to right, left to right, left to right, left to right, left to right and then you realize oh I have to save the change. So you again go over here that time should not be wasted. Start typing continuing the typing then reset index for resetting the index but that creates a new column and for that we understood that I can either drop a column or I can drop a row. If I want to drop a column axis is one and if I want to drop a row the axis is going to be zero in that case. After this I can also drop without using the axis. So in that scenario as I told you earlier what will you do if you want to drop without using the axis columns parameters can be used. Similarly if you want to drop rows without using the axis rows parameter needs to be used that's obvious and straightforward. Then round and numpy round we rounded off. So I told you numpy round will be faster and then the percentage column is converted to any data type whatever you want using the as type and this was an important master stroke over here that we calculated the result and see here the logic is simple so we can definitely do it using the lambda. So I say that percentage column I want to apply the lambda function. So every value of percentage column one by one goes into the variable per and then it checks is per greater than equal to 40. Yes, return pass else return fail. That's how it is computed and the same thing can be done with the help of userdefined function. So the name of the function is calculate result. You can give a call like this. But in such cases lambda functions are advisable. after that was the end of the session. Okay. Now what am I doing? Closing this and I'm kind of just for the reference renaming this as old pandas. Okay. So right now I'm naming it as pandas continued and old pandas. Both of them are available. I'll import the pandas library and we will read this data. We will read this data. Rest of the things I guess we don't need it. But both these data sets are read successfully using this. Now uh the thing is I am planning to look into the this I say info. Now if you look into this there must be ideally 9,994 rows everywhere. Then there are no missing values. That's a problem. That's a problem. And rest all things are good. Fine. CH missing values are there. I'm not filling the missing values. But what I want to know is if you look into the code, what do we see? Okay, I'll just save. Not yet. So now uh what I want to see is uh what is this data about? Number one, this data set is about categories. Categories like office supplies, technology, furniture. Three categories are there. I know that all of these are US cities, countries only US throughout. Every customer who ordered a product. It's like an Amazon data, right? Every customer who ordered the date product has a customer ID and then customer name, date on which he ordered, ordered ID, we don't need all columns. I'm not explaining. Postal code of the customer. Product ID whatever he ordered. Product name. Quantity ordered. The customer belongs to which region? US has four regions. Central, East, South and West. East, West, Central and South. There is no north in US. Central. Then after that we can see there are three segments in here. Consumer home office and office supplies. Then shipping date, shipping mode. There are three four types of shipping mode. State to which the customer belongs. There are around 17 subcategories. Three categories were there. There are around 17 subcategories. discount offered, profit, quantity ordered, sales, return on sales. We don't need all the columns. So, neither am I getting into discussion of this. Okay. So, this is what we have. Now, what I want to do, you know, what I want to do is I want to find out there are 17 subcategories. So, let's say paper is one of the category. You have binders as one of the category and so on and so forth. You have 1 2 up to 17 subcategories. And what I want over here is the average sales of. So I want to find out what is the average sales of paper, what is the average sales of binder, so on and so forth. What is the average sale of this category? Do you understand how you have to approach this problem? Not in code, but do you understand how do you how should you approach this problem? If you have to solve this problem, let's say in Microsoft Excel, do you understand how will you have to do it? Or at least the logic part. You will have to first of all filter the data for paper and find out the average sales. Then you remove the filter. Then you have to filter the data for binder. You find out the average sales. Remove the filter and keep on doing this till the last category and find that. Or another awesome suggestion is the pivot table. And third one is the group by. The third one is the group by. Let's understand. So now that you understand the concept, I am going to make you understand how does the group by works. Then we will get into the pivot table. This is the best example to understand group by according to me. Maybe according to you it is not but trust me it is easy. So just like I said you assume this column to be subcategory. Assume this column to be subcategory. So what will you do is there are three subcategories X Y Z XYZ. So uniquely you have X, Y and Z. You separate out all the X records. Separate out all the Y records. separate out all the Z records then that is what is called a splitting is done till this point of time splitting is done then you need to ask which function to be applied if the user says apply the sum function then 10 + 40 50 20 + 50 70 30 + 60 90 and then you combine this x50 x50 50 here, y 70 here, zed 90 here. In our case, we asked the function to be applied was not sum but average. So 10 + 40 upon 2. 10 + 40 is 50. 50 upon 2 should be 25. 20 + 50 is 70. 70 upon 2 should be 35. 30 + 60 90. 90 upon 2 would be 45. So here 25 35 45. If someone would have said no no you got to use the minimum function then minimum of 10 and 40 would have been 10. Minimum of 20 50 would have been 20. Minimum of 30 60 would have been 30. So this should have been 10 20 and 30 respectively and in this way it could have been applied to any scenario. Am I correct? Right. This is called as the group by. This is called as the group by. Yes. So now that you say you understand it, I am going to copy the link of this and I go to the next one group by and this is the link for it for your reference. Now you know let us keep the problem initially simple. in this data frame. Uh what I want to do is I want to find the average marks of the students who are recommended yes and recommended no. I want to find the average marks. So on what column will I group by? On which column will I group by? I want to find the average marks of student as per recommend. So before that I would like to show you the syntax. One is this part pf dot group by columns dot. So I will have to specify whichever columns I want followed by the function name and the columns. I'll explain you what it is. So I say I want to find out the average marks as per the recommend column. So I say DF dot group by recommend. DF dot group by recommend. And after that what do you want? You want the average marks right? So you use the mean function. And what do you want to see? Marks. No no marks is nowhere there. Uh research score. Research score marks. So you got to write that. Do you get that? No. In fact what does it say? It says that aggregate function failed and if I cut it and put it here you get this. So this is one way of doing it. I'll talk about it don't worry. or I say df dot group by recommend and aggregate average of this. Okay, you see the same output. Now pay attention what I exactly did. So I'm saying this is something which I usually use. I like to use this but this is also one way of doing it. It's your choice. Number one I say group by recommend. Okay. So we are grouping by the recommend. Here also the same after that I say I want to see the research scores average. So average marks of the students who passed is this in the research course subject and failed is this and here I say ag. Ag stands for aggregate. So which aggregate function you want to use? So I say I want to find out the research scores average. So that is what you see over here. Now whichever you want to use you can go ahead. The first one according to me is more common. However, however I want you to look into the format of the output. The format of this output and this output. So this output one is in which format or what is the data type of the output? And coming on to the format two, what is the data type of this output? If you want the data type of this output as well as a data frame, then this research score can be just put in double columns and you'll see the same output. Now you can compare all the three. So this is the one which I usually like. This is the one which I usually like. So always have a habit of putting double square bracket. Okay. Now if I copy this, you know, if I copy this, come down and average marks of the students per recommend. Now research score also. And I also want project score. Is that allowed? Yes. So this is the average mark score by the students in project score subject for not recommend and recommend. And this is for research score not recommend and recommend. Those of you recommended pi pivot table to me we will come to that as well. Don't worry. Pivot table as well we'll be looking into. Right now I'm not looking into it but we'll be looking into the pivot table as well. Right? So that's one very important thing. After that now that we understand this uh you can actually change the function. You can go for sum function. You get the total marks. You can go for min function. You can go for max function. In fact, you know what? You can also directly type over here describe. So you all know that when I say df do.escribe, this is what all you get. This is what all you get. when I say df.escribe which is nothing but count mean standard deviation and all the quantiles from 0 percentile to 100 percentile. So for research score if you see I am getting count mean standard deviation and all the percentile still here and similarly for the project score also we are getting the same thing that's one of the good thing. So you can do that conveniently over here. Okay. So that's one thing that you can think of doing. Then the next thing is coming to our super store data set. That's a problem because uh over here what do we want to find out is the what I want to find out? I want to find the average sales by each subcategory. So subcategory is correct. Yes. Sub- category and sales is sales s capital. Let's run it then. What is it saying? Aggregate function failed. How is equal to mean? D type is equal to object. These things can happen. Check the D type. I am finding out the average of sales. So whenever you want to find the average the sales column must be either integer or float. Whenever you want to find out average it must be either integer or float any one of it. Right? Please look into this uh what is the data type of the sales column object. Right? So I will you know this is where your role comes right. SS of sub category is equal to SS dot subcategory and as type two what should I make it uh looking at this value we can make it as float right yeah it's a point value is there so I will make it as float So I say float. You need not write float 64 only. Float is also fine. And then I'm looking into the D types. So it says value error. This line only gave an error. Now this code is fine. It says that could not convert string to float. Uh-huh. Why? Oh yeah. Not subcategory, you know. Sales. Sales. Sales. And then save it into sales only. Okay. Now it should run. 1 2 3. Run. Oops. Again. Not running. It says could not convert string to float. What could be the issue? It is a clear example of float. Yes, that comma has made it go into comma. Because of that comma, that entire number is not recognized as a float. It is recognized as a string. Rectify this. So, we understood what is the issue. In fact, if I say SS of sales and I just look at a sample of 10 values, looks everything good. Another sample good, good. Here is where we see the issue, right? The sample is randomly picking any 10 values and this comma, right? So what I want to do I want to get rid of this comma. Okay. So how to remove the comma. All right. Let us say you know let us approach this problem as if we are completely new to pandas and there is no one to help us. So what will you do? You will go to Google first and you will say remove comma from a number in a data frame. Remove comma from number in a data frame. And see there is a stack overflow thing which is helping us out. So this also suggests me but I'll go to stack overflow. And it says it says use the replace function. Okay, I like it. I was anyways aware of it. But I'm trying to show you how do you use it. So it says that you use a replace function and this was the code for it. So in our case, this is sales. This is sales. So sales is data frame not data frame SS. So in this sales column first of all str will make it a string from object and all the string functions will be available. See I'll show you that when I say str and I say dot see all the string functions are available. You can see over here all string functions are available right whichever we have studied in the string chapter and this is where I say let me replace commas by nothing commas by nothing and now I say ss now that's done Okay, done. And then now I'm looking at samples once again just to be double sure. Let a fourdigit number come. I don't see a comma. Huh? Everything looks good. Because everything looks good. The data type is still object. But now I think let me try out the code to convert it to float. This works. Now the sales data type is float. And because it is float, now I can find out this. Does that work? Yes. So we had a glimpse of data cleaning as well. We had a glimpse of data cleaning as well just because of this problem and these all things are very very important in the same way. Now I can find out the average sales per category also I told you there are three categories that's what I get. Now if I want this average sales per category in the descending order then I will apply the sort values function after this. So what will I do is this entire thing I'm going to put in a bracket and then I say sort values and sorting the values by the sales column. Ascending is equal to false. You see the difference? Sort the values by sales column. Ascending is equal to false. So it came in the descending order. What if you know you know if I say do you understand this question highlighted one I want to find out the sales of sales by category subcategory. Right? For example, you know, most of them didn't understand. Consider where we have students from first standard, second standard right up to 10th standard, right? And in each standard there are four groups. Uh the red group, green group, blue and the yellow. Here also the same four are there. You have that houses now. Red house, green house, blue house and yellow house. I hope you all are aware of all these things in the school. Ah you have the houses. Now what I want to do is I want to know what is the average marks secured per standard per group. So what is the average marks of the red group in first standard? Average marks of the green house in first standard. Average marks of the students in the blue house of first standard. Average marks of the students in the yellow house of first standard. Average marks of the students in the red house of second standard. So on and so forth till the average marks of students in the yellow house of 10th standard. Now did you all understand the question? What am I expecting over here? Now imagine this to be your category. You have three categories, right? You have you assume this to be category and assume this to be subcategory. So I want to know category subcategory by sales. Ah, so I will just put category and subcategory and you get this thing. So you know this makes it more clear that okay inside the category furniture these are the subcategories and these are the average sales of that. Inside the category office supplies these are the subcategories and this is the average of that. Inside the category technology these are the subcategories and this is the average of that. Right? Did you all understand how easy it was to do? I just increase one column. So you can do further more group by if you have further sub subcategories inside this also. Such guy are you all clear with this? Right. Is this exactly what is called as a pivot table according to me? It's no this is not truly a pivot table. I don't deny to you. If you recall in pivot table you have some rows you have some columns and there you fill in the values right? So based on something over here based on something over here you fill up these values. But here you have all of these things over here and the values are here. So I can you know now look at this I'll tell you I'm copying the same code pasting [clears throat] it down and then I am applying the unstack. Now do you think it's a pivot table? Yes, this is a pivot table. This is a pivot table. Pivot table will always have one column on the rows. It will always have another column on the columns and the values will be dumped inside it. That's how we understand a pivot table looks like. Some of you may say that we see some odd things like N and N. See if the category technology has a subcategory accessory then that is the sales of that. Obviously if the category is technology how can the category how can the if accessories is going under technology? How can accessories go under furniture and office supplies? It cannot. So it's saying not a number. Not a number. That means missing values. That means missing values. If there was a value, it will come. If there was no value, it will not come. Like for example, if you see in over here, if bookcases is in furniture, bookcases cannot be anywhere in the office supplies. If machines are in technology, machines cannot be anywhere in office supplies or furniture. Therefore, for machines in office supplies, machines in furniture, it should be NN. Let's check it out. You see, but for technology, it's there because machines were in technology. Do you understand why this is getting printed this way? So, we did that using this and unstack function. Remember this unstack. Now if you think that Dan sir this code was too complicated uh do we have a direct thing to do this? Yes I can do achieve the same thing and that can be done using a pivot table function something like this. Now pay attention I'll show you. Output is same. Point number one output is same. How is the code different? Please understand very carefully. What did I say? The index must be category. So that's the index. Obviously it's a data frame. Then this is the index. That's category. Then it says the columns must be subcategory. So obviously these are the columns and that is subcategory. And the third thing which it says the values must be sales. Which one? Average of sales. The values must be average of sales. So all of these are average of sales. See whichever you find it easy you go ahead. I usually find the group by easy. So I prefer that but if you find the lower one of pivot table easy you can use that you know in this case in this case according to me the best way to represent this is this one not even the unstacked one not even the pivot table this is the best because you don't get confused because here there are lot of nans as well right so in such scenarios it's not advisable to go for unstack or pivot this is the best Now if I want to find the names of the students who secured more than 60 marks in the research score subject. So obviously I will have to select the research score subject and in the research course subject that's going to select that entire column of the research score subject and what I want to know I want to know the students who got more than 60 marks. So if I do that you know if this is greater than 60 it will give true. Greater than 60 true greater than 60 false true true false true false true you can just have a look at it. When I do that this is what you get. But you know what I'm not interested in that. I am interested in finding out where all this condition is giving true. So in the data frame in the data frame look for this condition in the data frame you look for this condition. So I'll copy this comment out this in the data frame you look for this condition and I get this. But what are we interested in? I clearly said what I want. I want the names. I want the names of the students. I want only the names. So this is giving a data frame. So from this data frame I want to select the name column. So for selecting any column I said that you write in square bracket this. So I can write it ahead of this only. I can write it ahead of this only. And this is what we get. And what is the data type right now? Numpy data frame series pandas series. Right? Yes. From this series, do you all agree that these are the indexes of the series and this pink box is the values of the series? Index values. Index values. We have studied all of this. Yes, I am interested in the values. I am interested in the values. So, I will further say dot values. See, I'm showing you all step by step. I'll further say dot values. Now, what is the data type of the output? Could you tell me it's a list numpy array? Array series. It is a numpy array. It is a numpy array. Always remember whenever you see array return in a pandas type of work it will be a numpy array. That's why I proved it right. It's a numpy array. Okay. You understand that it will be a numpy array. So from the you know numpy array I have to convert it to the list form. How can I convert a numpy array to list? using the function tool list. See numpy array is converted to list using the function tool list or you can also use two underscore list. Ah it doesn't work but it works in some cases. It will work in Google collaboratory. So I'll not do that. In Google Collab it works. Both the versions of it work. So you can do it this way. That's one way or you can say. So see this is also fine. You can type cast it to a list or take this entire thing and use the tool list function to do it over here. Okay. So we understand this and now let's say I want to find out the students who secured more than 60 in the uh research course subject where this right these were the students who got more than 60 in the research score subject and more than 70 marks in the project score subject. So this is what I want. So this was the code for finding out the students who got more than 60 in the research score subject and for the project score I will replace it like this and that was for more than 70. Yeah, these are the students who got more than 70 in the project post. But I want to look for both. I say in my data frame, I want to look for condition number one and condition number two. I want to look for both. That's my first condition that gives me true or false. That will be pasted here. That's my second condition which gives true or false that will be pasted here. And now you get the combine output. Then if I want to find the names of all the students whose name starts with alphabet H. Then I'm going to write the code like this. DF that's the condition and then DF of name which will select the name column dot str will convert this object to string for time being so that all the string functions available one of the string function is starts with I say starts with h these These are the true things. Look for that boolean condition inside the data frame and that will give it this. And then further if you only want to find out the names, you know, you can write square bracket name dot values dot tool list. That's okay. Now let's get started with that. We are now getting into the last part of pandas which is concatenating two data. Thanks. Have a look at this. I have created a data frame one. How to create? I am not going to talk about. We know that we created a data frame one. We created a data frame two. I say concat the data frame one and data frame two. So how is it concatenated? A B and C D. So if you see this is my let me use conventions. So I said this is the data frame one and this is the data frame two. So how did it actually happen is data frame one and two when you say concatenate they are getting concatenated side by side. Okay, this is what we see as the default behavior and as you can see A and B these all things came and inside this C and D these all will come but obviously because for A and B you don't have the corresponding values of C and D they are not a number and same goes over here now obviously this is not an ideal case of concatenating but I just wanted to show you how the concat function works Okay. Now, one should never use concat without the access. One should never use concat without the axis. And the default over here is access equal to zero. So, if I actually do that, you see the same thing because the default is access equal to zero. So I can say it's the or okay and if I say access equal to one now you see something meaningful what has happened is data frame one data frame two now you can see this is your data frame one and exactly to the side of it is your data frame two and this is what axis is equal to one does right truly sideby-side concatenation and not row wise. So depending on the use case you can think of concatenating it. Now you'll be like but where do we actually need concatenation? So let's consider you are Adani electricity. So today you have a customer base of 50,000 customers in a particular location right? So many people are using your electricity connection. This is the status as of April 2025. In May, new buildings were built up and they actually started and another 100 customers or let's say another thousand customers have started consuming that. So, Adani might be already having its customer database of these 50,000 customers. customer 1 2 up to 50,000. So that could be considering the customer name, customer CA number, customer pin code, customer units that are consumed, etc., etc., etc. Now another thousand people have purchased that connection. So that thousand will be appended below. What are we doing? Isn't that concatenation? Yes. So these all are the practical places where concatenation is usually helpful. As simple as that. I'm talking about the merge function. PD dot merge function implements a number of types of joints. One to one join. Many to one join. Many to many join. All three types of joins are assessed via an identical call to the edmer merge interface. The type of join performed depends on the form of input data. So I'll explain you everything right now. Let's get started with one to one join. I say they are similar to columnwise concatenation. So I'll explain you with the help of a simple example. It's very very helpful because in real life you will never entire data into one table. For example, consider Amazon. how it might be storing its data. Customer data in one table. All the orders given by customer data in another table. These are linked. Then its employee data in another table. Then its products data in another table and its uh transit data in another table. And all of these are linked together. So tomorrow if you want to figure out which customer has given which order you may have to refer only two tables or you may have to refer three tables at once and get from this your result but you got to create a join of this and that's exactly where we need this. So I am going to create a data frame PD dot data frame and let me create a data frame of employee over here where I say we Oh, we have employee and let's say the employees are Bob. Then another employee, let's say, Jake Lisa and Sue. Okay, that's done. And then after that, let's say the group to which they belong. Accounting, engineering, engineering, HR. Yeah, that looks good. And yeah, that looks good. In the same way, we have another data frame which is PD dot dataf frame. And inside that let's say I have employee which is Lisa, Bob, Jake and Sue and let's say the higher dates are 2004 8 12 and 14. Let me print data frame one and data frame 2. Right? Now I say df3 is equal to pdobs and I'm showing you df3. Now let's understand this code. Right? So these two data frames are created. As far as the code creation is concerned, I'm least bothered. You know how to create a data frame. I'm just not interested in explaining you this. But I'm interested in explaining you this. Whenever you are trying to merge, whenever you try to merge, always remember that merging is possible if and only if there is at least a single column in the data frame which is common. Can you tell me between the yellow and the pink data frame, is there a common column? Is there a common column? Yes. And I'm sure you are talking about employee. Okay, great. Now see how it happens. Bob belongs to accounting group and Bob is hired in 2008. Bob accounting 2008. Jake belongs to engineering group. Jake is hired in 2012. Jake engineering 2012. Lisa Lisa 2004. Lisa Engineering 2004 Sue belongs to HR and Sue is hired in 2014. So Sue HR 2014. So by default it considered the common column and did the mering. There are many other use cases of it. Okay, that's easy. Then I get into the next one which is many to one join. So I say many to one joints are the joints in which one of the two key columns contain duplicate entries for many one. The many one is for the many to one case. The resulting data frame will preserve those duplicate entries as appropriate. Let's see that. How does that happen? I am creating a data frame for and this is going to be consisting of group and let's say group I'm saying accounting engineering HR supervisor let's say Carly Gos and I'm just printing data frame three data frame four and merging this. Let's understand what have we done. First of all, you can see I have a data frame employee. I have a data frame over here [clears throat] about the group of the employees. Do I see a common column? Employee group hire date group supervisor. Yes, group is a common column. Then I say Bob belongs to accounting group hired in 2008 and for accounting his supervisor will be Carly. Okay. So that's one to one mapping. Jake belongs to engineering group hired in 2012 and for engineering the supervisor is Goyo. Jake engineering go again one to one. Lisa belongs to engineering hired in 2004. So for her also the supervisor will be Guyaido. Yes. Many people are mapped to one supervisor and hence it is called as a many to one join. Next thing you can verify whether I'm done many to one comes to an end right I'm sure everyone understood it please acknowledge Perfect. Thank you my 57% of the audience. Okay. So now the next one is many to many joint. Many to many joints are a bit confusing conceptually but are nevertheless well defined. If the key column in both the left and the right array contains duplicates then the result is a many to many merge. Then I'm saying data frame file is equal to I'll say I'm going to create a data frame and inside that group is accounting let's say and once again let's say accounting, engineering, engineering and then I'm planning to go for HR and HR and then I say skills let's say math spreadsheet coding Linux spreadsheets organization Okay, printing the data frame one, data frame five and three and five merged. Let's understand this. So you can see employee data frame you can see groups department the common employee group group skills. So common column is group. Now how is it many to many? Let's see. Bob belongs to accounting group and for accounting group one needs to have the skill of math and spreadsheet. So one too many has already happened over here. Bob accounting 2008 math. Bob accounting 2008 spreadsheet. Le Jake belongs to engineering gor. And for engineering you need the coding skills and the Linux one to many. Jake engineering 2012 coding and Jake engineering 2012 Linux. Lisa also belongs to engineering and for engineering again the same skills are required. So now you see many to many as well right because many people like see Lisa and Jake both are belonging to group engineering and for engineering you have requirement of two skills many domain end of the story. Now let us talk about the confusion that one can have at times. Here we see that group and group are common. Imagine there was one more column called as the employee. So now the problem is this employee and employee would also be common and this group and group also be common. So when you say PD do merge data frame three and five it will be grouping on what? It will group on this or it will group on this confusion. So to solve such scenarios they say that if this is the kind of thing then in such scenarios what you can do is you can think of using our parameter on. So I say on is equal to whichever column you want to group. So let's say if I'm saying uh DF1 uh let me come in let me first check it let's say data frame one and data frame two right so here when I say PD do merge data frame one and data frame two you see this but if I want to specify the common column is employee now assuming that there is one more column so I can always say that on is equal to employee right. So that on parameter specify the column on which to join simple okay now I'm creating one more data frame I call it as dataf frame three and that is is equal to pd dot dataf frame name Bob Jake Lisa Sue Salary 70812090 Do you know what is a problem? Tell me a common column in this scenario. Which is the common column in this scenario? So you people were able to figure it out because you are humans. Correct? How will the machine figure it out? How will the machine figure it out that this employee and this name are the same as per a machine? It's a confusion. So you have to tell the machine because if I directly come over here and I say PD dot merge data frame one with data frame three it gave an error. So you know now understand the way I see about this. See listen to me that's my code right? Give me a moment. So, so now think that You know what? Now just like I am looking at you. Let's say this is you. You are looking towards the screen. You're looking towards the screen. So we all know that this is your left hand and this is your right hand. So to the left hand you have data frame three. To the right hand you have data frame sorry left hand you have data frame one. Right hand you have this. So what did you all just say that here in this you consider employee and here you consider name right? So you got to specify it that way. So I have that parameter. Left on is employee, right on is name and see now it works to the left side to the left side DF1 which is employee to the right side in DF3 consider and then do the job. Yes, you do see one problem over here that employee column and name column two times have come but that's okay. I think that's not at all a big thing because if they are coming for two times you can always say that I want to drop the name column and you only have the column of interest name column it's a column right for dropping a column you need the access parameter and it is done left on and the right Remember this parameters. Okay, understood. Acknowledge quickly. Now we are almost at the end and then I will keep the scope open for questions on pandas. So I say I am creating a data frame 6 as PD dot dataf frame and that's going to be name as Peter Paul and Harry and I'm considering food as fish beans bread name as Mary Joseph and drink as Right. So we have these two data frames created. I just want to talk about the merge function. Have a look at this. The common column is name. We know that I didn't do anything. Peter eats fish but because no Peter is available here. No record came. Paul eats bin but because no Paul is available here. No record came. Mary eats bread. Mary drinks wine. So Mary bread wine came. Joseph drinks beer but no Joseph here. So no record comes. This earlier we never used to see this that okay there is only one common record and only that comes because I had taken the records of the data frame in that way so that you are focusing on that concept. But now I want to tell you that if it is not available it does not come if you are doing an inner join. In fact, I can do the same thing. And here you have the parameter how is equal to inner. Same record and same output. So that how parameter can help us out. And you can create any other join like left join, right join, outer join and the inner join. So I'll copy this DF6 and 7. So common column is name. Peter eats fish but no Peter available. So Peter fish not a number. Paul drinks bean but no Paul available. Paul will slaughter. Mary bread marry wine. Mary bread wine. Joseph beer but Joseph not available. Joseph beer food not available. Outer join it will take all the records. So depending on the use case you can take any one of the joints. I'm sure you have already studied joints at some point of time. So this is what I wanted to say over here. Now welcome to the world of data visualization. We are going to start the very first visualization. Now ideally when it comes to data visualization it will make more sense when we visualize the data for a data frame but the basics needs to be understand in a simple way that is what we are intending to get started right now. So just to give a heads up, I would like to now get started with visualization with Matt plot lib. So, Mattplot lib is a multi-platform data visualization library built on the top of numpy arrays that multib is a multiplot multiplatform library built on the top of numpy arrays. That's the very first important thing. So over numpy mattplot lib is there. One of the most important features of the Matt plot lib is its ability to play well with many operating systems and graphical backends. And this cross platform everything to everyone approach. has been one of the greatest strength of Matt plot. Okay. Now this visualization library was built as an alternative for the MATLAB users. So there was earlier a software which is called as nat lab this is how it used to look like and it's a paid software for your kind information it's a paid software it can be used for all of this and today also it's available but it is paid that's the logo for it and it was very popularly used for data visualization And this is how the software used to look like. Matt Bloodl was originally designed or It was written as a Python alternative to the MATLAB's plotting capabilities and you will see much of its syntax and functionality is similar to this. However, I would say in the recent years, Matt plot lib has begin to show its page. You have newer tools like plotly, ggplot and seaborn have emerged that offer modern capabilities. How a word? However, or I would say making mattplot li feel clunky and old-fashioned. Still I am of the opinion that mad plot li you know opinion that we cannot ignore matt plot li as Uh well tested crossplatform graphics engine. Okay let's go ahead. So now let us get started with the library. Now first and foremost thing let's import the libraries we need. So import mattplot li right. So piplot piplot is the library for creating static animated and the interactive visualization in mattplot li and you will need only that. I mean since last 15 years I have been consistently working on these this pipelot only. So that should be more than sufficient. Now the next is plt dotstyle dot use and you can use any style. So what all styles are available? Well, if you want to actually find out I want to print all the available styles, then you have plstyle do. So these all are the styles which are available. So styles are nothing but the themes, right? So any visual can be plot. I'll show you. So this set of visuals if you use the style instead of BMHs classic will look like this in dark background will look like this and so on and so forth. Similarly this will look like this like this. So these all are the available styles. You can see how each of these visuals are looking like now which one can I use? Well, let me use this type seaborn colorlind. You can choose any one of it. It's your choice, right? You can use any one of the style. It is your choice. By default, it is classic. So I was saying by default it is classic and there we go ahead. So now I set the style as this. And now I will kind of recall numpy. So there is a function of numpy which is called as lind space. So if I say 0 comma 1 comma 5. So what it does is it returns five uniformly spaced numbers in the range 0 to 1. So if you now try to find out 0.25 - 0.25 -25.25 this minus this.25 25 1 - 75.25 everywhere it is 0.25 25 only if I say 0a 1 into 10 equal parts you can find out the difference right this minus this.11 this is this is also.111 same thing or if I say 0 to 7 in 10 equal parts see this minus this is 77 this is 77 this is This is also 77. So what am I doing is I am saying 0 to 10 into 100 equal parts. 0 to 10 into 100 equal parts. So I have 100 values. I'm sure all of you understood this. You know 0 to 10 divided into 100 equal parts. I don't want to print it. This is just for your reference. And the y will be the sign of each of these values. Sign. So what is sine of zero? That would be y. Sign of this, sine of this and so on and so forth. So these are the sign of the 100 values. I'm going to comment on this also. But you understood what is x and y. Can I assume that all of you have understood what is x and y? Yes. Now I see let's get to the plotting. So I am talking about the plot function. So plot function says I need plt.plot plot x value y value color marker line style label. So x value, y value, color of the line. Example can be given like this. Marker of the line, line style of the line, label of the line. Let us check it out. I say plt.plot plot x and y. So I'm saying that x consist of the 100 values. Y consist of the sign of those 100 values. So it's very simple. We have to plot this. I got this. You see over here I got this. I just gave these two parameters and one can understand this really well. Then after that I don't like this. You know what is this? This is basically saying that this figure is saved at this address. I don't want it. So I prefer to put a semicolon. Semicolon will suppress that. See that has not come. But if I don't put semicolon it comes. You see this comes and here it doesn't come. Another way is you can avoid that semicolon is by saying plt.show function which is mainly used to display the plot. plt dot.show is used to display the plot. See that also hides it. So plt.show is a very common way of going ahead whenever you have to display the visuals. Okay. Now the size of the figure is very big. So I say this has a function which is called as a figure function which has a parameter which is called as a fixed size. If you make it 1 comma 1 that means it will keep the size of the figure as 1 comma 1 becomes 1 in + 1 in. Can I make it 3a 1? You can. Can I make it 1a 3? You can I make it. 3a 3. We can. And I think 3a 3. Okay. Can I make it 4a 3? We can. I think 4a 3 is looking good. Or maybe if I say 4A 2. Yeah, 32A 2 is looking good. I think I need not you know in my notebook 3A 2 looks like a good size. So I'll stick to this. You can play around with this L bus. I'll get in your correct values also. Okay. Now plf figure by default it gave the blue color by default it gave the blue color you can also set in the color over here so I set color as red and red color. So color which I can give is red, green, blue, black, white or I can also give color in short forms. Anything is possible. So this is red. If I give the color in short form K. K is black. K is black. W is white. R, blue, black, white. There are many colors available. Okay. Then you have the line style. So line style can be dash. So I'll tell you what all are this. Dash means a solid line. Dash dash means a dashed line. This means a dotted line. And this means a dash dot line. You see it has become a dash line. If I say colon, it becomes a dotted line. Okay. So that becomes the thing label. We'll use it later. We'll use marker in this case. Well, marker will also be interesting marker. Marker is equal to O. That means dots marker size is five marker size is two.5 05. So let's remove the line style. And now you see the marker style over here. Right? So marker size if I keep it as 0.5 you see the dots over here. If I keep it as 4.5 you can see the dots over here. If I make it 2.5 you can see the dots over here. And if I set the marker as D that would be diamond. Observe carefully there is a diamond and this can be easily understood when you plot a scatter plot. See this when I go for pl.catter scatter same code size of the marker if I keep it as you can see this if I'm keeping the marker as D that's a diamond if I keep the marker as plus it would be plus everywhere there. So what all markers can you put these are the various types of markers or you can also refer this Or we can also refer. There was one very good image. I just looking for that. Yeah, this can also be considered. You can use the marker style of your choice. That's upon you. So you can plot it. You know I'm not talking about scatter plot and all. We'll be looking into all of these things later on. But this is what we see. Now let us say I simply say pl.plot plot x y and then I say plt dot show correct now here in this case just like x and y I can always say x versus numpy dot cos of x. So sine and cos both. So when sine and cos both are getting plotted, you see these colors are automatically being chosen. But I'm not able to understand which is sign which is cos. So what I can do is I can put in over here the line style as this and here I can put the line style as this. So I know that a solid line is a sine view and a dash line is a cos view. Or I can also think of using the label parameter that this will be labeled as a sin x and this will be labeled as a cos x. Does it come? No. If you have to display it, you'll have to use the legend function. You have to display it. Then you will have to use the legend function to display the legend. legend function is used to display the legend. Okay. Now if I use the same code and inside this legend you also have the parameter location of the legend which I can set it as any one of this. So if I say location has to be upper right it will get shifted to the upper right and you can give any one of these values. I would recommend you to always put the location value as best. It will automatically choose whatever is the best place for itself. Automatically choose the best place. So it will not be overritten anywhere. Now what am I doing is you know see here at the top I'm going to save it as dashion. So dshion is equal to plt.figure. So this figure which you see right now on the screen is saved or not saved is yeah saved into the object called as dian. See I printed dan. You see that? Now if I want to save the figure then I have to say dion.png and you can just give in these parameters or you can skip it as well. So let's say you can directly say print figure saved as dian.png is it saved? I can check it in my local environment. Dian.png. Yes. And if you want to save it as a particular DPI value or something. This is the one DPI 300. This is the regular one. This is the one. This is what all you have now. What are formats in which you can save it? Right. So that can be looked into person dot canvas dot get supported file types. So you can save your figure in any one of this format. I did choose the PNG which is most popular. Don't ask me about all of the others. I don't use anything other than PNG but EPS this this this these all things are understood this is also understood raw is okay SVG is okay and WP rest all of the formats even I don't know and I don't need it as well okay now the next thing that we are looking into is when I say listen to this. This is important. This is let's say this is your figure. You want to divide it into two parts. H two equal parts. then I can say this is two rows and one columns inside which I call this as panel one and this as panel two. So this is the first one where I want a sine wave to be plotted. This is the second one where I want a cos to be plotted. That's my par one. That's my panel two. How do I do it? It's very simple. You have to use the subplot function. So plt dotf figureure creating a new figure plt dot subplot two rows one column and currently I'm talking about panel one. I want to plot x and y. Similarly, I say plt subplot two rows, one columns, panel two. Have a look at it. Two row, one column, panel one, panel two. or when you have two rows, two columns. This is panel one, panel two. Par three and par four left to right top to water. So I say pl dot figure el dot subplot 2a 2a 1 in this first panel x and y in the 2 comma 2 comma panel 2 cos of x in 2 comma 2 comma panel 3 I'm saying tan of x and I forgot about panel four I didn't write anything about this it will be empty or if I forgot about panel three this will be empty have a look at the code and then we have this control s now this is mattplot li style of doing it you can also say over here that let's say you I can also think of it. Listen to this very carefully. I can also think of this that this entire thing is my figure and I am having access. I am having access. This is row 0, row 1, column 0, column 1. So this is 0 comma 0. This is 0 comma 1. This is 1 comma 0. This is 1 comma 1. So say how do you do it? This is a different style of doing it. The function which we used over here is subplot. I'm doing the same thing but different. So what am I doing is figure axis is equal to plt dot subplots. Is there a difference in the function earlier and now? Do you see a difference in the function? What is the difference is there? What is the difference? Same. No, there is no difference is there or not there in the earlier function and new function. The earlier function is there in the chat box and the new function is there here on the screen. Yes. extra s subplots that's correct so this is subplots don't forget that this is the subplots figure is the name of the figure and this pink all things which are written is the things so I say over here figure is the object ax is the axis of the object a x 0 0 1 1 0 1 1 are this. Now I say over here a x 0 0 will be sin 01 would be cos 1 0 would be tan and 1 1 is tan hyperbolic and you plot it. This is another style of doing it. Whichever is convenient to you, you can use it. This is the objectoriented style. That was the mat lab style. Whichever is convenient to you, you can use it. Now I think anything else other than this, yes, I want to also talk about few last simple simple things which are left out. Uh if I say plt.plot X and Y, right? And right in this there is no label on the X axis, no label on the Y axis. I can say plt.title title sine wave plt.x label x-axis not x-axis I would say x only and plt doy label sin of x. So you see I have a title X and Y and I can also say over here font size is 20 to make it more bigger and that is what you have in over here. Perfect. So X level Y label also done. Then after that we are good to get started with the I think this is done. Yeah. Then we are good to get started with the data set on which you can do all of these functions. Yes. Now we get into the actual use of the charts. Which chart to use when? So I was saying which are to use in which scenario. This is what we are going to look into right now. So we say histogram. What is a histogram? Histogram is nothing but a frequency chart. So if you look into this Very nice. Yeah. Here. Now what do you understand when you look at this histogram, right? Why do I call it as a frequency plot is because we see that this is the tallest. What do we understand? If you look into this number which is the height of trees. So we understand there are 50 or I can say most of the trees in the data set are in the height range 250 to 300. So that is one of the inside that we understand most of the trees will be like this right. So that's a histogram. Now if you look into this, the next one is a pie chart. We all know that pie chart is mainly used when there are categories. But I would say as far as pie chart is concerned, one should always avoid it. We're going to plot all. But you know the problem with pie chart is if I ask you this is big or this is big tell me it is so similar it's very difficult some of you may