Applied Data Science With Python Full Course 2026 [Free] | Python For Data Science | Simplilearn

Key Takeaways

Hey everyone and welcome to the applied data science with Python course. So every time you shop online, watch a video, book a cab or use a banking app, data is being created. But the real power is not just in collecting data. It's in understanding it, cleaning it, finding patterns and using it to make smarter decisions. And that is exactly what this course will help you learn. In this course, we will take you step by step in the world of data science using Python. So you don't need to feel overwhelmed by big terms like numpy, pandas, visualization or statistic. Here we will break everything down in a simple and a practical way. Here's what we will be covering. First we will understand what data science is, why it matters and how Python is used to solve real world data problems. Then we will explore the complete data science workflow from understanding a problem and collecting it to cleaning, analyzing, visualizing and preparing it for machine learning. Next we will get into important Python libraries like numpy where you will learn how to work with arrays, numbers, mattresses and even calculations. After that we will move to pandas, one of the most useful tools in data science. You will learn how to work with series and data frames, filter data, sort data and handle missing values and clean messy data sets. We will also cover practical data operations like working with datetime values, text data, summary statistics and data exploration. Then comes one of the most exciting parts, data visualization. Here you will learn how to create charts and graphs using Mattplot liib, seaborn and plotly so you can understand data visually and present insights clearly. We will also touch on important math and statistics concepts like distributions, outliers and linear algebra which form the base of many data science and machine learning techniques. Finally, we will see how all of these skills come together to prepare data for real world analysis and future machine learning models. By the end of this course, you will understand how to take raw data, clean it, explore it, visualize it, and use it to find meaningful insights in Python. So, let's get started and learn how data science actually works in the real world. Also, if you're ready to take your skills in Python and data science to the next level, check out Simply Learns data science with Python full course. So, this course is perfect for anyone who wants to master Python programming for data analysis, visualization, and machine learning. So you will learn how to work with key Python libraries like numpy, pandas and mattplot lip for data wrangling and analysis. You will be diving deep into data visualization, feature engineering and statistical methods that are crucial in the field of data science. Plus you will get a hands-on experience through real world projects like sales analysis, marketing campaign analysis and more. Upon completing the course, you will earn a course completion certificate from SimplyLearn which can boost your career and showcase your new skills to the employers. So check the description below for the link and start your data science journey today with simply learn. Before we get started, here's a quick quiz for you. Which Python library is used mainly to work with tables, rows, columns, and structured data? Is it a numpy, b pandas, c mattplot liib, or is it dplotly? Please drop your answers in the comments below and let's get started. >> Okay, so we're we're just on lesson one. Uh just going to get basically a road map of what we're going to study. Um so this this should just highlight different topics. I'll just quickly touch on uh what we intend to cover over the our next um kind of 12 sessions that we have dedicated to data science. So the first topic which we will study tonight is uh really going to be lesson two but here it's kind of topic number one which is uh introduction to data science right so just focusing on uh some foundations like some definitions what is data science what packages are going to help us in data science just quick overview of those before we dive into them further and then uh get an idea of the process that we would use in uh solving data science problems like what kind of process will we follow typically um I think that'll be a big uh highlight is that process because we will follow that process um as we dive into more data science problems. So that's going to be in the intro that will segue us into the next couple of lessons which will cover um some really foundational packages for us inside of Python namely the numpy package and the pandas package. So, numpy we're going to get to today. Um, and that's going to cover basically the usage of numpy in data science, why why we use it, um, some syntax, uh, some examples, some practice with it. Um, so we'll get familiar with numpy and then uh that will lead us into the next lesson which will be on pandas. So that's another Python package that we will study and get a lot of practice with. Yeah, pandas is going to be incredibly useful for us because uh as the name suggests like data science where we're going to be working with data most of it's going to be structured and pandas is going to help us work with that structured data and explore it and slice and dice it and do a lot of fun things with it. So, Pandas, uh, if you're a big like Excel fan, Microsoft Excel, and you work a lot with spreadsheets, there's a lot of similar functionality that pandas kind of provides, but we're going to be doing it in code, right? So, it's going to be very exciting there. Um, so we'll do numpy, we'll do pandas. That will segue us into um, how do we kind of tell a story with our data. So, you know, we will be talking about data visualization, for example. That's one big way to tell a story, which is to focus on drawing graphs, building out plots. That's always important. It's it's useful in two ways. One, so we can get a handle on what the data actually means. We can visualize it. That's helpful. And two, it helps us externally tell a story, right? Uh so if you wanted to share any insights, um usually a picture is pretty powerful. And so drawing those graphs um going to be incredibly useful. So we're going to study some different code and different packages that can help us build visualizations uh of our data. So that's going to be pretty useful and that's in um lesson five. Then following that will be some math and uh stats fundamentals. So we can't get away from the fact that there is some math and stats involved in data science of course. So mainly just want to get some background on it. Um I don't mean to scare anyone and and think that we need to be experts on math and statistics. That's not really the case. Um but there are some basic concepts that I think we should be aware of and I'll do my best to break those down. I I have a math background. I went to graduate school for math. I studied graph math in undergrad. So I like to think I have a good grasp on it and can help you guys too. It's again not the intention to be an expert in math but just to go over the basic concepts we would need to be able to work with our data right so so there will be some basic math and stats fundamentals we'll cover because it's necessary you can't get away from it when you're talking about data science so we'll we'll cover that we'll also go into probability um because we also will need to talk about how our data is distributed and a little bit about probability not not too in-depth but just enough to cover the basics of what we would need to um again describe like how data is distributed, what that means. So we'll look at some different distributions and kind of uh how to interpret those. That'll be more important for us than the nitty-gritty kind of math details, I think. Okay. Finally, we'll get into some interesting topics to round out our math and stats detour. So if you look back at this, we kind of take a little bit of a detour and and this will be lesson six, seven, and eight. that will be a little bit of a detour and then in covering math and stats and probability and then advanced stats which really will cover um hypothesis testing. So that'll be interesting. It'll give us a way to kind of think about um how do we formalize doing things like AB testing or how do we formalize doing like a hypothesis test um between like a control group and an experiment group. So, how do we quantify that? Which is an important aspect of data science and something that people usually care about, especially with products when you're rolling out new features, usually roll it out to a smaller group uh and compare it to kind of a baseline group and you want to quantify those differences. So, we'll talk about that in that advanced statistics uh lesson. Then we pivot to the last two lessons of the course which will be I think our my two favorite which will be getting us uh really doing a lot of data preparation for modeling. So we're not covering modeling in this course because that will be in the next one in machine learning. But uh lessons 9 and 10 which are here data wrangling feature engineering will be incredibly useful for getting our data prepared for modeling. So we need to learn how to clean it up, how to um slice and dice kind of raw data, how to create new features out of existing data, um how to fill in missing values, how to do scaling, how to do grouping. Those are those kind of slice and dice operations. So, a lot of things in those two lessons that will help us prepare our data for modeling. Okay, so that's kind of the gist of those last two lessons, just getting our our data prepared and cleaned up, ready to be modeled off of. Okay, so if I were to summarize that, we have kind of our intro, which we'll cover today. Numpai we'll cover today, that's an important package that will help us um work with our data. Pandas as well that'll follow that. Then we'll go into building plots. Then we'll go into math and stats detour for a few lessons and then finish up with getting our data working with our data right getting it prepared for modeling. So a lot of exciting topics. Hopefully you guys, you know, feel excited about that too. A lot of really cool things that we'll cover. Um but that's that's kind of the that's kind of the road map. Okay. So we're moving on to lesson two. All right. So lesson two, our intro to data science. So, we're going to get just some background on what data science is, the what what it is as a field. Um, and then some of the um again some of the processes, some of the packages that we'll need, some of the the tools that we'll need to work on data science. So, by the end of this lesson, we want to be able to talk about uh data science in general. Just define what it is and what what we would be doing in data science. like what is if you're a data scientist what kind of things are you doing mainly want to focus on the process too like what are the steps to solving problems systematically that we will end up following Jab it's the ebooks I think it's the ebooks oh you may need to you may need to update your course okay um and then we'll take a look at the so I was saying um that we will basically go through the steps to solve problems I want to definitely highlight that process then we want to look at some of the packages that are going to help us do different things in data science like data analysis, data manipulation, data visualization. So what are some common packages in the Python ecosystem that we will use to accomplish data science? And then uh lastly uh there's a description of some plots. Uh I'm going to uh skip those because we're going to have a whole uh section on visualization where we'll go through those pots uh in more detail, but kind of briefly I can I can show what those look like, but we have a whole section dedicated to building those pots. So I'm not concerned about going through them right now, but those are at the end of this lesson. Okay, so let's dive into it. So what is data science? Okay, so what is data science? It is we had to put a definition to it. It is a multidisciplinary field that uses scientific methods, processes, algorithms, systems to d this is the big part of it derive meaningful insights from structured and unstructured data. So that's the that's the big part of it is obviously data is in the name data science. So our goal with data science is to derive meaningful insights from data. And so that can involve many things. That can involve building models which we will learn how to do later on when we get into machine learning. That can involve building out visualizations. That can involve um doing uh hypothesis testing. That can involve many different things. but we're deriving some type of insight or finding out something interesting about our data that we have through different methods. So, it's it's kind of I always like to say it's the science of working with data, which is um you know kind of a weird way of saying it because that's obviously the name data science, but it truly is the science of of extracting insight from data. um is is this field and there's a lot involved in it that blends a lot of different disciplines together. It's why we say it's multi-disiplinary because um we kind of bring together multiple uh aspects of math and stats, computer science, domain expertise um to be able to derive those insights. So there's, you know, things there there's ideas and concepts borrowed from uh uh science like like doing hypothesis testing. There's um obviously math and stats. A lot borrowed from there. There's a lot borrowed from visualization and analysis that we that we use in data science. But we kind of blend that with technology, right, with computer science. So being able to use Python is a big deal. that will be our our language of choice for doing anything data science. That's why we kind of reviewed it in the beginning. So we'll certainly use Python. We'll certainly use different tools within Python to process data. Those are data processing tools. So we kind of blend those together to form the field of data science. So th those scientific methods along with um tools working with data in from technology like Python blend those together we kind of get data science. So where do we see data science today? Um there's a a lot of different um applications and I'll try to describe some of them to you which is uh for example like wearable devices. Uh so think about like Fitbits or Apple watches. they're always crunching data from their sensors, right? So, there's some type of biometrics that are captured and sent over uh the internet essentially to basically um allow us to do some some type of analysis or some type of derivation of insight from that data. uh and then we can kind of visualize that with some graphs or some meaningful metrics um so that the person wearing that device can make some sort of decision. So there's some type of insight derived some and usually some type of algorithm being applied to that maybe a model's being built working with the data that's captured from the wearable device or we're just plotting that data or we're summarizing it in some way but making it really useful to the to the end user to make a decision off of. So we're deriving insights there um from all of the data collected in the the wearable device. So that's kind of one application. Um, search engines use data science to uh personalize results or offer recommendations as people type in their their queries. So um essentially like you have suggestions, right? Like and these could be based on your previous browser history. um all kinds of data like your cookies or your what's trending in the in the world or like your region though you probably have noticed this right when you're searching on on search engines you get these suggestions those recommendations are kind of powered by data science right so data is going into that to derive some insight that hey this is what we should s should suggest and that's uh it gets surfaced there one of the things that we'll look into down on the road when we get into machine learning which is the next course after this will be recommendation systems. So how do we actually build those models that do recommendation but in order to build those you really need data uh so that's where we got to start is working with data to to um get in a position to model off of it. uh finance we see um usage of of data. For instance, um there could be like some type of model that's built to uh determine um a loan decision. So there could be the application and then there can be additional data that's gathered via the details in the application and then that all that data can be brought together and kind of analyzed through the use of a model that could predict yes we should give this loan or no we should deny this loan. So again, you're deriving some type of insight from that data to make a decision, right? To make some sort of loan decision in finance or it like we see a lot of fraud decisions made as well like is this transaction fraud or not fraud? That's another big use case of data science. So based on the data of transactions, are they fraudulent or not fraudulent? Okay, a lot of other applications. Those are just a few and and as we go along throughout this course, we'll really study a bunch more and get into some more use cases as we go along. This is just a preview. But what I want to go to next is kind of the process by which we attack data science problems. So we know that we should be deriving insights from data. That's what data science does. Okay. All right. Let's talk about the process. Okay. So the process is outlined roughly here, but I'm going to take us through every single step. But if I had to break this down into basically a handful of key steps, it mostly starts at here, which is kind of initial problem formulation, which involves collecting data. So usually you have problem formulation. So you need to know like what are you know do we want to build a recommendation system for movies? Do we want to um figure out if transactions are fraud or not fraud? Like we're coming up with some problem that we want to solve and then going out and collecting data that might help us uh analyze that problem further. Um so we'll talk about that. Then comes the kind of uh data preparation phase I would call it. So data prep meaning that we are preparing our data for modeling um with with that goal in mind is our goal is to actually build some sort of model to tell us something to help us make a decision or derive some insight right and so um we will do a couple steps I'll describe what they mean in a moment of doing some data preparation in order to get ready for modeling then comes the actual modeling uh phase is the actual modeling phase which involves um building, training, evaluating all of that. And so there then comes the modeling phase and then comes the actual um like uh deployment of of the model meaning that we um use it in the real world uh to to bring our insights into an actual uh system integrated into actual system so that those decisions can be can actually be made and used by by an end user let's say uh I believe you can call a data science process even if no model is built step four five. Um, you can, but so I think that's fair is to say like if even if you're not building a model, so you're you're thinking just steps one, two, three, four, or even just one, two, three, I think that's fair. But most of the time in this course or in this program, we're really going to be building models. Um, so I want us to have this in mind is like even if we stop here and don't do any model building, this is what we have our eyes to is the ability to the ability to uh model if we want to. Okay. But I think it's a fair point is that it could be, you know, it's a fair point that you could just imagine getting your data together as really like kind of a data science process. Is feature engineering a feature that will use the so so feature I'm going to describe what feature engineering is in a moment. I'm gonna go into that in further detail in the next couple slides. So just hang on when I when we say feature engineering we'll talk about it. I was thinking for AB testing and prior where there's no model. Would you call this a data science project? That's up for debate. Um AB testing sometimes that falls under analytics sometimes that falls under data science. We'll we'll talk about doing AB testing. So even when there is no model. So I think it's fair to say it's part of data science. Um which would kind of just be steps 1 2 3 4. So yeah I I see where you're coming from. I see where you're coming from. It's it yeah traditionally AB testing is more like experimentation analytics. I I agree. It's not traditional data science but we will cover it because it's an important aspect of of data science with with the idea that the you know the part of deploying a model and seeing what the results are would be like an experiment right it would be we think of it as an experiment where I build some model see what the results are on on a different group than a kind of a baseline control group that's not exposed to that model. So it's important for us to know what it is. You could argue the actual process of doing AB testing is not really data science. But if you add in that context of usually um people are interested in using it in conjunction with building a model and and exposing it to users as an experiment. Does that make sense? All right. Let's go into each of these steps and talk a little bit more about them. Okay. So usually things start in the beginning with a a problem definition which is a goal uh or a question that will be addressed through collecting and analyzing and deriving insight from data. So that's the very first step and usually this this is actually something that you would work together with other uh colleagues usually to come up with. So usually you may come up with this with like a product manager or with another engineering group or you know something like that to come up with a problem like hey we need to build a recommendation system for our users to get better recommendations for their movies or for their shows or for you know their products on our website. So, so there usually is going to be a question or a goal, you know, or hey, we want to come up with a a sales forecast for the next um three quarters given the data we already have for this quarter um or the previous quarters. So, you know, usually there's you have to start with a problem. So you have to start with a problem definition um that you want to you want to address and and honestly that goes handinhand with the next step which is once you have a problem in mind you then have to collect data around that problem. So you have to gather the relevant data sets which could also involve working with external partners to do that. So maybe you have to work with data engineers to help you go out and collect that data or make that data accessible. you may have to work with uh you know you may have to do it yourself and go and gather historical data that can help answer that problem. So it sometimes this is up to you. You have to uh go and and go out and collect that historical data or that data that's relevant to your problem. That is totally possible. But sometimes you're also working with external partners like a data engineer to make that possible to to go and collect the data. But of course, the key word here is relevant, right? The data needs to be relevant to the problem that you're solving. And that can be a challenge. So, I know these are listed as the first two steps and they seem pretty straightforward, but they can be the most challenging at times is defining the right problem and collecting the right data, getting that available um not always trivial. But in this course, we'll usually assume that we have these two things. We'll usually assume that yes, there's a problem. We we know we want to address for practice purposes. We have that and then we also have data already available for us. We don't have to go out and fish for it and collect it and scrape it from somewhere. We'll assume that's already been done and we're just using the data that we have available. So in this uh in this course this these two steps will usually be done for us mainly just for so that we can practice all the other steps but but in the real world that would usually be you know someone you're working together to formulate a problem with with external stakeholders and you're also gathering data either by yourself or with with the help of um maybe like a data engineer or someone like that. Okay. So again uh these two things we'll usually have in this course which leads us to the next phase which is that data preparation phase. So this is where I said um we need to clean the data and explore it a little bit. So usually this process can take some time. So this can be a very time consuming process but the typical tasks we're doing here are getting a handle on any missing data. So figuring out a strategy to handle missing data, we'll talk about that. Um how to identify and handle outliers, we'll talk about that. Uh maybe duplicate data, inconsistent data that that doesn't make sense, we can identify that, we can get rid of it or have some strategy to handle it. So getting a handle on cleaning up our data is going to be an important task. And that might take some time. That might take some time to work with our data and kind of do some we have to learn how to do the proper code, clean it up, get it to a good state. And then once we do that, we can start to explore it a bit to gain insights. And this is where we'll usually use um visualization at our at our disposal. So maybe we'll build some graphs to quickly visualize, get some patterns, see what see what the data looks like, figure out if there's any relationships in the data. That's where we'll learn about visualization that will really help us. So when we say explore, we're mostly talking about building um graphs to help us kind of tell a story about what we see in that data. Assuming that it's been cleaned up, right? Assuming that we have it cleaned up, we've removed all our missing values, outliers, inconsistent uh values, we now have a clean set of data, we can start to explore it. So, usually um doing visualizations, it could also be it's not only visualization, but it could also be summaries. So, maybe it's really useful to tell a story like what is the average for all these users or what is the average sales for the last few weeks? Um what is the median sales? like those kind of statistics might be really useful summaries to tell a story about the data. Okay. Okay. So, we have our steps here. Problem definition, data collection, cleaning, exploration, first three steps. Okay. And again, we're going to like as we go along, we'll we will we will deal with uh we will deal with these um problems. We'll do have actual examples that will deal with this process. So we'll see this process from end to end many times as we get into our examples later on. Bronze, silver, gold. So usually those are like data engineering terms to refer to uh this the basically how clean the data is. Um so like bronze is kind of like the rawest form. It it's usually like data that has not been aggregated in any way. It's usually pretty raw. It hasn't been cleaned up in any way. Silver might be cleaned up but not really aggregated. So silver might be like we've removed missing values, we've done some we've removed outliers. Uh so silver is a bit cleaned up, but it's still kind of raw. And then gold would usually be like our final transformations have been applied to it. Like maybe we've done some averaging. Um we've done some transformations. So usually th those are terms that you see to refer to the different like stages of quality. Gold being like the highest quality like the final data set. We don't really, by the way, we don't really use those terms too often. I think you see those terms a lot in like data engineering. Don't really see them that much here just because the assumption is that we're always going to clean up like our models aren't going to be good unless we get to a gold state, right? Unless we clean up our data, unless we do the right transformations, that makes sense. Like our models really aren't going to be useful until we reach that state. So our we're always going to be pushing to like clean up and get to a good state. Okay, so we have one, two, three. Let's look at the next few steps. So once our data is clean and we've explored it a little bit, this is step four, which is feature engineering. So this is the other kind of data preparation step that I talked about. So feature engineering, what is that? It is a um creation or transformation of new features. So you might ask, what is a feature? A feature is just a variable that is an input to a model. Okay, a feature is just an input. So I want you to think of like in an Excel spreadsheet, a feature is something like a column. It's like a it's an independent variable like a column that we would use to build a model off of like that will be one input. We would call it a feature that's going into the model uh as an input. Okay. So feature engineering is the process of kind of building new features and those can be by doing simple transformations like maybe we do some scaling like dividing by 10, multiplying by 10. Those are simple scaling we can do to features. We could do more complex transformations like doing a linear transformation to it. Um we could take a square root, we could take a logarithm, we could take an exponential. Many different transforms we can do to our data to create new features. and and sometimes that makes sense to do that. Sometimes we don't need to do that much feature engineering, but that's something we will get a feel for as we start to do examples is like when does it make sense to do feature engineering and when when do we not have to. Um so feature engineering will be more of an art than a science honestly. And um we're going to do plenty of examples where we do feature engineering to see what kinds of transformations we typically will do. Everything we're talking about will be part of the current data we're working with. Is it always performed? No. Uh we don't necessarily always do feature engineering but it is a step like we can and we should evaluate if we should. Okay. So will we always do it? No. But we have the ability to do it and it is a it's a step worth calling out because it can be very valuable to do. So again, we haven't learned how to do that yet. So it may not make that much sense to us, but I'm calling it out as a very important step in the process and as we start to do examples later down the road. Um we're going to come back and spend some time on feature engineering because it is important step. It is an important step often to do it. Okay. So this was step four and this was um data prep. So draw a line here because this is all like the data prep and then these couple of steps here are all modeling oriented. So once we have our data prepared, we've cleaned it up, we've done exploration to determine what we should keep, we've done some engineering to do some scaling or transform the features that we have, we can now build a model. And so this will all be um something we will learn in our next uh we will learn how to build models in our next course on ML. But just calling it out as you know that would be the next natural step is once we have our data cleaned up to derive an insight from it we may want to build a model off of it. And so that will involve um kind of defining a model training it on this data that we've prepared um as as step number five. And then step number six will be kind of an evaluation step of um determining if we have a good enough model by evaluating it. you know this by the way this process isn't necessarily linear in the sense that we may iterate here and go back and you know repeat these steps. So we may go back and forth and keep repeating these um by building a new model determining if it's good enough repeating and repeating and repeating that process and we may even go all the way back up to here and build some new features if if the model isn't doing that well either. Um that may be possible. So by no means is this process always linear. We may repeat especially this model training and building and evaluation steps. We definitely can repeat these back and forth um until we kind of converge on a good enough model. That's something we'll discuss when we get into modeling. I don't want to go into details now. But that is suffice to say like we can iterate those uh steps quite a bit and we can spend a lot of time on it for sure as a as a data scientist. Okay. And then, oh, by the way, there's there's one more. So, there should be a step here on deployment, which is kind of mentioned here, but I would argue it's its own it's its own step. But once we have the model um done, then we can kind of deploy it. Now, deployment means a lot of different things. There's a lot of different ways to do that. We won't get into that till much much later in in the program. It's it's not really going to be a focus for us at the moment. We're mainly going to focus in on all of the data preparation steps and then all of the modeling and then leave this part till kind of the very end of the program. Okay. But it it is important part like we need to make you once we build a model we do want to have it be useful in the real world. So we need a way for it to be integrated into existing you know existing systems which can be different there different ways to do that. Okay. All right. So let's go back and circle around to Python. I know, you know, we spent some time on it already, but just to reiterate that Python will be our friend here when we're doing data science. So it'll it's going to be the preferred programming language for anything data science and that's true in the industry. Python is widely used mainly because it has so many great packages to help us work with data namely numpy and pandas which are the first two we will look at and then it has many others to help us build models like scikitlearn which we'll get familiar with and then it has others to do visualization that we'll study. So it basically has packages that do most of the tasks for us that we're interested in doing. So that you know that's why we'll stick with Python. It's really great for data science. So, we've talked about this before. It why we why people prefer to use Python is because it's open source interpreted. It has so many great packages that are oriented for data science and and can help us do data science really easily. A lot of people used to use R to do data science, but people it's been a shift um over towards Python because of its flexibility. Um, Python can integrate with other systems pretty easily whereas Rs R is more difficult to use. Yeah, R is like another it's like a scientific analysis language, you know, it's used in a lot of like statistics. Um, a lot of statistics people like using R, but uh for doing data science, it's almost exclusively done in Python. So there's really no you don't see R too often. I've I've really never seen it. I've only seen Python in the industry, so no worries about the R. Historically, R has been around uh R has been around for a while, but um Python is by far and a way the most used uh data science uh language I've for sure. Okay, so I want to briefly tell you about some of the packages that we are going to study in in our course that are in Python that we will use to do data science. So I just want to briefly talk about them and then of course we're going to have um a couple of lessons dedicated to going into those like numpy and pandas and all the visualization libraries. So the first one is numpy. So numpy is uh short for numerical python. So that's that's why it's called numpy the numerical python and it is a python package for doing computing basically scientific computing uh using these uh array structures that numpy has created and so many things are built off of numpy arrays and the ability to operate on these numpy arrays. So, NumPy came around and um created these multi-dimensional arrays which are essentially like matrices um and and also had a lot of different um computing uh tools around the the matrices and arrays that so many other packages are built off of. So, we're going to learn pandas. Pandas is built off of numpy. So is uh mappl which is for plotting and so many other packages are built off of numpy. So it's a really foundational package for working with data because data will be stored in numpy arrays. The numpy array is the kind of foundational data type of numpy and and so many things work with numpy arrays. Okay. So numpy will that's going to we're going to have a whole lesson dedicated to numpy coming up next. But numpy is going to be the first place we're going to start just because it's so important for working with data. It's multi-dimensional arrays are so useful for storing and manipulating data. So it's it's pretty important. What is a forier transform? It is a transformation of uh data into like a signal basically like a signal transformation. So you extract you go from like a a basically like a time series into like a frequency series. It's used in signal processing. Okay. So the second package that we will study, so we'll start with numpy. We'll start with that today. So right after this lesson, we'll dive right into numpy and start working with examples of the numpy arrays. But right after that is the library pandas, which we'll spend a lot of time with. Pandas is a library built off of numpy. So it it depends on numpy and it basically comes around and provides a more structure uh to manipulating data. So if you're like I said earlier, if you're familiar with Excel, pandas has a lot of functionality that mimics what you would do with a spreadsheet. Basically like structured row column data um is is what pandis excels at. So, pandas is going to be a really fundamental package for us to manipulate data that's structured in kind of a row column matrix format, but it's built off of um numpy. So, it uses numpy under the hood to do all the manipulation, but pandas provides its own data structures to kind of put data into almost like a spreadsheet format that we can manipulate. Okay. So really pandas is going to be really really powerful for us to manipulate data and we'll use it all the time. So I if anything coming out of this course you guys will be pandas experts if anything else. I mean of course you'll learn more than that but I think you'll come away as being really really good um users of pandas and numpy for that matter but but certainly pandas. So we're going to study NumPy first and then we'll have a lesson dedicated to pandas right after numpy. So we'll have a lot more to say about it but I just wanted to kind of preview that you know it's a really important package in the data science ecosystem because it helps us manipulate that structured data that's in like a row column format like a table. Okay. Then another package is the sci package which is um short for scientific python. It is another open source library that's built on top of numpy. So it uses numpy arrays as its underlying uh data structures to do the manipulations. Scy contains a lot of um scientific formulas and a lot of um scientific computing tools that we'll use especially when we get into hypothesis testing. So it contains a lot of like z tests, t tests, distributions, things like that. So it's it's tailored for that. It also has things like the forier transform as well. Um it has different linear algebra manipulations as well. So sci will be really useful uh when we get into our hypothesis testing and AB testing. It has those kind of uh those distributions that we'll need to do our tests like a like a student t test or a z test or those kind of things we'll we'll use scypi for. So really important package. We'll see later we do hypothesis testing. Um, another one that is going to be useful from time to time is the stats models package. So, it is one that um basically has a lot of statistics oriented things. It it has um some basic models in there like like linear regression or logistic regression. We will generally favor a different package to do those kind of models. But just calling it out that stats models does have some useful stuff when it comes to doing statistical testing. So there are some like kiquare tests or ANOVA tests that we will borrow from stats models that sci um we can borrow from stats models. So we will use it when we get into hypothesis testing as well. So these last two, so sci and stats models are two packages that we'll use when we get into AB hypothesis testing. Okay. So that brings us to scikitlearn. Now this is going to be our primary package for doing machine learning. So this will be one that we'll build all of our models and machine learning off of when we get into our machine learning course. So we we won't really use scikitlearn in this current course, but when we get into machine learning, uh it will be our go-to package to do all of our uh machine learning with. It is a fantastic fantastic library that's been developed over over years to contain all the basic models that we would ever want to build. Psycharn's really awesome. Um so it can it can build models for so many different use cases and it's a really easy package to use. It has a really nice interface, really easy interface. So we will see that later on when we get into our next course on machine learning. But just calling it out that is a very popular uh data science library scikitlearn. So when we get into our modeling we will use scikitlearn. When we do our data prep manipulations we'll be using numpy and pandas. Finally for visualization for visualization we will be using a library called mapplot lib. So it is kind of the foundational Python plotting library that borrows inspiration from uh from from mat lab. So if you guys have ever used the mat lab plotting um it's actually very inspired by that hence the name matt plot um from mat lab but it's it's going to be our main tool for using uh for for building graphs. Okay. So, um it's a foundational library for building graphs. Almost every other library that does visualizations is built off of this one, built off of Mattplot Lib. So, when we get into our visualization course, we will come back and do uh we will come back and talk a lot about MattPod lib and practice with mapp quite a bit. What is that course called? Uh machine learning. Our next the next course is called machine learning. Okay. And then another uh visualization library that we will lean on heavily is the seabor library. This is one that is built on top of mattpot lib. So matt mattplot liib is kind of like numpy. It's the foundation and then a lot of things are built on top of it. Seabour being one of them. Seabour being one of them. And it basically has just better aesthetics. It provides better not only like better aesthetics than just basic MPA lib. It also has more scientific kind of plots and more interesting plots than the regular ones you get out of the box with map paw lib. So it has really interesting histograms, file plots, heat maps. Um it can do statistical error like confidence interval bars. So it just builds better plots than than basic map liab. Map pod li is very basic. It can it's really easy to use. you know, you can build a lot of plots with it as we're going to learn, but Seabor is really nice. It makes things aesthetically pleasing. And so we'll also use Seabor from time to time. It's another plotting library that we'll get some practice with uh when we get into visualization. And another one of those is Plotley. So we have Seabor and Plotley, both built off of Mattplot Lib in order to do plots. Now Plotley's specialty is for building interactive graphs. So when you build a plotly graph, you can actually um it'll pop up in your web browser kind of like Jupyter notebooks do and you can click around in the graph and mark down points. You can zoom in, you can zoom out, but you know you can zoom in, you can zoom out, you can do a lot of interactions with potly. So if you want to build an interactive graph, potly is a good package. Again, it's built off of uh Mattplot liib. So we'll get some practice with potly. So, so these three we're going to practice when we get into visualization. Seabor, my plot li is the basic foundation. Seabour and plotly both build off of it. We'll get some practice with all three of those when we get into visualization. Okay. So, what I wanted to do is take a fivem minute break. I know we've been going for a little over an hour. Um, I want to take a fivem minute break and then when we come back, we'll finish up this lesson. It has some uh plot examples. I'm going to just briefly go through those mainly because we're going to have a whole lesson dedicated to visualization. So I don't want to spend too much time on on uh plotting just yet because we will get to that. But um after that we'll get into numpy. So we'll get uh right into numpy and some of the code which will be nice. So the rest of these slides just go through some plots that we will be building later on when we get into our visualization. So uh just wanted to brief briefly go through those just to show you some of the different types of plots we'll do. So the easiest kind is basically a line plot that connects different points. So this would be like if we were plotting out something over time like a stock price or uh a sales value over different quarters or or weeks temperatures over time something like that. So basic kind of plot we'll be able to build that no problem. Um, we can even mark different points on those. That'll be easy to do with mapp or seabourn or potly. That'll be really easy to do. So again, we'll we'll show you how to build these with code later on when we get into visualizations, but just showing you the possibilities right now. Uh, scatter plots. We'll do these which have um different points kind of uh scattered throughout on on uh two axes here. Um, this is usually helpful to figure out how the data is kind of um maybe clustered together or figure out if there's relationships between two variables, like if they tend to trend the same direction or in the opposite direction or if they're kind of just distributed all over the place. So, we'll be able to build scatter plots that'll be helpful. Area plots that show like cumulative areas on top of each other. We'll be able to show that. um that'll be pretty easy to graph for different maybe tracking total sales uh over successive quarters showing different contributions of categories. We'll be able to do uh so we'll be able to do area plot basic bar plot we'll be able to do uh again these all of these examples were built using map plot lib so we'll be able to do that but they have equivalent versions in pli and seabour so again th those are built off of mapplot liib so uh we can even put grids in the background to show uh to to kind of um assist the viewing of it uh to to give an idea of where the different points are in the grid. So that will be easy to do. Um histograms. Now histograms are going to be extremely useful for us. We'll build histograms a lot because they will help us visualize how data is distributed which is extremely important to know you know is it kind of distributed like this in this picture which is kind of like a bell curve or is it flat? Is it does it have kind of two peaks to it. um knowing this distribution will be extremely useful to us. So we will often build a histogram that kind of looks like this. So histograms will be extremely useful. Uh we can build piraphphs um which show different percentages. Uh so you know there may be certain situations where that makes sense. We're telling a story of our data. It makes sense to use a pigraph. That'll be easy to do. The again these are all just examples of what's possible. we have to show you how to build these and we will when we get into the data visualization lesson which is what this kind of note says at the bottom once we get into that lesson we will show you um how to do the code to build these. Okay. So just to wrap up this first introductory lesson, we uh have shown you what data science is, which is kind of the uh extraction of insight, deriving insight from data and we have a bunch of different packages going to help us do that. We also have a process which is going to help us do that which is usually defining a problem, collecting data, doing data preparation and then doing modeling after that. So you know basic foundations at this point what we're going to do is now go into uh numpy. So we're going to start with numpy and then go into uh pandas after that. So we're going to start studying these packages are going to help us do some of these different tasks in data science. >> Okay. All right. So let's talk about numpy. Remember, numpy is the open-source library that is used for doing you know that is used for doing math and scientific uh computing on uh basically these arrays. So um we are going to take a look at the numpy array object as the first thing that we'll look at. Now the numpy array object behaves very similar to a list. The so we learned about lists in our previous course and the the numpy array is very similar to a list. We can slice it like a list. We can access elements like a list. It's ordered like a list but it's a lot faster to do mathematics with the array. And it comes with a bunch of built-in functions like mean, median, mode, all these special things on the array that we don't get with a list. For instance, Python lists do not have a notion of an average. You can't calculate the average of a list without doing a manual calculation. So, but a numpy array has a mean function that comes built in that we can uh take the average. Numpai has like an average function that we can take of an array or a median or a mode. So, arrays are really advantageous to work with inside of numpy. So let's take a look at some examples of a numpy array. So in the in the first uh cell here I want to point your attention to two things. One is that in order to use numpy we import it. Do you see how we import numpy and we do this thing called asmpp. Now what this is is we do an alias. This is called an alias. Alias numpy as np. We basically shorthand it to np which is an industry standard. So anytime you're looking at code and you see np do something that is short for numpy. So in the industry if you you know everyone is going to shorthand numpy to np. That's just that's just what people do. So as is the way to alias and import so that when we use numpy in our code we don't have to type out the full word numpy. We can just do np. So that's why you see np here is because uh and really throughout our code we use np. You see it all over the place. It's it's a shorthand alias for the numpy package that we're using. So uh we are importing this package meaning that we are going to use it in our code but we are aliasing it to np. This is the this is the industry standard to do. Most of these packages have a nice uh alias to them like pandis will have an alias uh mattplot li will have an alias um just to make it shorter. All right. So, the next thing I want to point our attention to is building a numpy array. So, notice that we can build this numpy array by doing np.ray. So, nparray np.ray um builds a numpy array object. And what we're passing in is just a list of data. Okay, so we have a list of integers that we pass into this MP array which will build a numpy array out of this list. So numpy arrays can be built out of lists. They can be built out of tupils. They can be built out of other numpy arrays. There's many ways to build a numpy array, but the most common is to pass in a list to convert a list into a numpy array. So here we are uh building a numpy array from a list which is pretty typical. By the way you guys remember I said that I'm going to be writing a lot of comments. I'll share these notebooks in our Slack after after the classes. But I encourage you guys to do the same thing is to write comments in your notebooks. Okay? try to write comments in your notebooks to outline what the code's actually doing. How do you install numpy? Uh you just need to do so inside of a cell. Inside of a cell, you can run this uh command like pip install numpy. Try running that inside of a Jupyter cell. Yeah, this this command is not going to work for you because this is like a generic this is on um this is on Windows like a generic Windows command. But if you're inside are you inside of a notebook, Mariel? If you're inside of a notebook, just run this inside of a cell. It should install it. Yeah, that works too. You can open your terminal and do pip install. Um if you do that, you'll probably have to restart your kernel. No. So, so Collab comes with NumPy already installed. Yeah. So that's another advantage of of Collab is it already has that installed. We don't need to worry about it. Yeah. So So if if it says requirement already satisfied, um which is what this is going to say if I run this um it's because it's already installed. So it this means that I already have it installed. Yeah, you already have it installed. Yep. So in Collab it already exists. this one new cell and this command. If you can't get it to work in your VS Code, I really encourage you to to do Collab as much as you can just again just to get something that works because NumPi is already installed in Collab. So, there's really nothing you need to do extra. Thanks, Tim. That'd be great. That'd be great. Okay. All right. So if we so this builds going back to this this builds a numpy array off of a list. So if we run this code what's happening is we are building an array and storing it in this array variable and we can print the array. Now look at what the array looks like. It kind of looks like a list when we print it except we the way we can tell this is a numpy array is that it does not when we print it it does not have the commas. Notice that the data in there does not have the commas. And that's because it's being treated as a numpy array. So it doesn't have the commas. It's not at that point. It's now an array. It's not a list. So it it looks slightly differently. And you can even see when we print out the type that this array is actually a numpy n dimensional array, which is the foundational data type numpy. So this is an a numpy nd array which is the foundational uh data type of numpy. So we have created a numpy array and we now you can see what its type is is this uh mp and d array. Okay. Were you guys able to run this first cell? If you run it it does it's not going to do anything but show this. You should see this and then you should see it's printing out the the type. You should see those two things. And do we see how that this is creating a numpy array? So np.ray is how we that's the function we use to build a numpy array. And we're passing in a list of data to build that array. All right. Yeah, it might take a it might take a moment to start up the kernel. All right. Okay. What I wanted to do is go to the next cell and talk about how we can create some matrices essentially multi-dimensional arrays. So the this array that we've created so far is actually just a onedimensional array because it's it only has it only has one dimension to it. It basically has one list of data. But of course we would be interested in working a lot of times with multi-dimensional data because that's typically like what a spreadsheet has, right? rows and columns. So just to give you guys an example like numpy actually supports zero dimensions which is basically a constant. So a single number a single value is considered a z array. So a single value is considered a zero dimensions is just a single value. So if we built a numpy array and just passed in a single integer or it it doesn't have to be integer it could be float like 24.6 six, you know, whatever it is, it that would be considered a zerodimensional array. But we've already built a 1D array, which is just a single list. Basically, a flat list with uh so just a if we use a list with a list of uh I should say a single list of values is a 1D array. So we've already we've already seen that it is a scaler. Yeah, we would call that a scaler. Yes. Uh, good. Yes, that's true. Scaler. Perfect. So, a single list of values is going to be a one-dimensional array here. Now, what gets interesting is now when we do a list that has list as its elements. So, this is a list of lists is now a 2D array. So I want you I want you guys to see that how we we're building a numpy array out of a list but look at what the elements of the list are. They're actually lists themselves. So you see how within this overall list the first element is a list that is 111 that kind of mimics basically like a row. So you think of it as like each list each list is like a row in a matrix in a matrix. So this two-dimensional array is really like a matrix, right? So so this is an interesting use case where you know of course we could have more than just these two. So we could have a third list here that is like um four, five, six and that would be valid as well. So this would be basically a matrix that has three rows and um each each row has three basically three items in it. So we would think of it as basically having three columns, right? So it's like a 3x3 matrix, but it is two dimensions. It's a two-dimensional array. It has rows and columns at this point. So it has rows and columns. The two-dimensional array. Do let me ask you guys, do we see how this has two dimensions to it? It's a list of lists. So it has two dimensions. Does all the lists need to have the same rows? What do you think? What do you think would happen if Let's try it. Let's try making this a smaller dimension. Do you think this is going to be allowed? Let's see. So yeah, this gives us an error. So yeah, it you're exactly right. This will give us an error that the dimensions do not match. So this this is uh not allowed. But let's see if I do if I add in the six. This should now be okay. And there it is. It's now this is now okay. No more error. Okay. No more error. So yeah, it's still going to be an error if if the shapes do not match. So again, if we got rid of if we made this a smaller one, that's going to be an error. Um, and it's going to tell us that we are uh we have one dimension that is inhomogeneous, meaning it doesn't match. It's not the same. We have one dimension that's not the same. Its shape doesn't match. It's not correct. So therefore, we should correct that and make sure it is matching. Okay, so that is a two-dimensional array. list of lists. By the way, that doesn't have to stop there. We could keep going. So now we have a threedimensional array which has lists of lists of lists. So it basically has one of these matrices as each element. So see how this has basically an overall list. So this has each it has a list where each element is a 2D array, right? Each element is a matrix. So here is one of those matrices is the first element and then here is another 2D matrix that is the next element and this forms a threedimensional array. So if we print this out we can see that the we get this 3D array where this first matrix this matrix is the first item this matrix is the second item and on and on and on didn't understand how 2D is different from 3D does it do you see how okay so do you see how with the 2D basically we take this whole thing and that's just one element of the 3D this This 2D matrix is one element and then we have another 2D matrix as the next element. So we have matrices are now the elements of the 3D array. Whereas look at what's the elements of the 2D array. They're just lists. It doesn't have to be two. That's just the example we have. It doesn't have to be two. But by the way, you one thing that gives away the one thing that gives away the dimensions is how many of these brackets we have. So you see how we have two brackets and see how this has three brackets. Can I So what's an example of using a 3D array? Yeah. So something that uses a 3D array would be like a a batch of images. So let me give you an example. So an image is like a 2D array because it has pixels, right? It's basically an image is broken down like this that has pixels with whatever resolution. And so if we have a collection of those that is it's like we have a collection of these guys is a 3D array. Like if we have a hundred of those it's like a 3D array. Does that example make sense? Like a collection of images would be a 3D array because every image is a is a two-dimensional matrix of pixels. Does this does this example make sense though? Like this this is a good one. I think I'm glad you asked it because I think it's a good one for thinking about what a 3D array is. Every element is a 2D matrix. >> Okay, perfect. All right, perfect. So, just to recap this, we have the numpy array. So, we're we are able to build numpy arrays using the np.ray. And we're able to take a list of data and populate it into an array. and and then what we're going to do is just build off of this to learn how to manipulate that array and do different things with that array coming up next. All right, so let's go to the next notebook. Let's go to 3.02. Okay, perfect. So let's go to our next notebook. Okay, do you guys have the 3.02 notebook? Do you have it up? That's the next one we're going to do. So, take a moment to pull that one up. Yep, I see some thumbs up. Okay, so we're going to build off of that numpy array by taking a look at some attributes of arrays. So, so assuming we have an array, no matter how many dimensions it is, what are some attributes of this array that we can that are useful to us? Okay, so let's take a look at an example where again here we import numpy and we make we're initially making a 2D array, right? So we make a 2D array. So then what we're going to do is print out a bunch of the uh attributes of about this array and we're going to explain what they what they do. So the first is if we ever want to know how many dimensions an array has there's actually an attribute for that which is the which is called n dim. So if we just do end dim by the way we access let me call that out here is we access attributes of an object by using the syntax object dot attribute. So in this case we have our array is our object. So it would be like ie i.e. be array.shape would be an attri shape is an attribute of the array. Array is our object here. Okay, so that's our syntax to access different attributes. So the first attribute we're going to learn about is called in dim which gives us the number of dimensions. And so when we print that out, you can see what it's going to be. It's going to be two. And that makes sense. It's a twodimensional array. Okay. So the first is nim, which gives us the number. This gives us the the um number of dimensions which equals two in this case. Okay. And that makes sense. We know it's a two-dimensional array based on the fact that our elements are 1D lists. So it's it's a we have a list of lists. It's going to be two dimensions. Now shape gives us shape gives us the gives us the quantity. And so it gives us the basically the number of rows and columns. So in this case what this is saying is we have two rows and three elements in each row. So shape is giving us an idea of how many elements we actually have or what the shape of this matrix is. That's why it's called shape. So in this case we have a 2D array that looks like that kind of looks like this, right? where we have 1 2 3 and then in this example four 2 five. So we have two rows, two rows and we have three columns. Two rows and three columns. Hence we get a shape of 2x3. Does that make sense? 2 by3. Okay. So shape shape is going to be incredibly useful as we go forward because um there's a lot of times where if we have an array we actually just want to know how many rows and columns it has which is the shape. So the shape is a good attribute to know does the bracket define it as a 2D versus 3D array. It's it's the fact that it's it's two brackets defines it as a 2D array. It has two brackets here. It's a list of lists. This is 2D. It's the fact that it has So I meant to say I meant to say that the in this shape we're going to see two entries. Sorry. In the shape we're going to see two entries. So the the dimensions will match how many entries we have here. So if we have two two entries uh because it's too twodimensional that's that's what the truth that's the thing that so sorry Roberto I was wrong on what I told you it is when we have two entries here that's because it's two mentions not the fact that this is a two it's two entries in the shape by the way how can we get a third row we could just add in another list here right so if we add in another list that has three elements like um 7 8 9. This is now this is still a two-dimensional array. But what I want you to notice is what I want you to notice is that it's going to turn into this shape is going to turn different and this size is going to turn different but it should still be 2D. So see how this shape went from 2x3 to 3x3. So now I want to talk about the size. The size is the total number of elements total number of elements in the array. So in this case we have in this case we have 3x3. So we have nine total elements. So this will always be the product product of the shape. Right? Okay. Do we get do we see what size is? Size just gives us the total number of elements. Okay, total number of elements which is really just a multiplication of the shape. There's three rows, three items in each row. So a 3x3 is nine total elements. Okay, so that's the total number. The size is the total number of elements. Now the the dype is telling us what every member's type is. So maybe not that interesting, but this is kind of the default. So this gives us what each elements each element's data type. So we can um grab in this case they're all integers but they're in N64. Um so we can also grab how many bytes each one takes up which is the item size attribute. This is each element's uh memory footprint each element's memory size which is going to be uh eight bytes. And we can also get if we want to very rare we would actually need to access this but we can actually get the uh memory reference memory reference for the array data. So array.data, we we won't really ever need to worry about this, but this is actually really important for pandas to be able to access later on because everything in pandas is built off of numpy. It needs to manipulate the raw memory uh often in order to do different calculations with that data. So it typically will need access to that uh data attribute, but we generally will never need to know what that memory address is. Okay, I think the one that we'll use the most is probably going to be shape. We'll probably worry about the shape the most uh when we're working with numpy arrays. >> Okay, perfect. Okay, let me show you a couple functions. All elements in a shitty array must have the same data type. No, they don't have to. Just like a list, they don't have to. When we're working with data, they typically will, but they don't have to. All right, I want to show you a couple of functions we can do to manipulate the shape of an array. So the first one is we can actually reshape an array using the array.resshape function. So this is a function that we can pass in. So we can do arrayshape and then we put in the uh a tupole with the new shape. So in this case we're putting in uh 4, 3, which is to say we want to take this existing array that is a one-dimensional array. So notice this is a 1D array and we want to turn it into a two-dimensional array, right? That is 4x3. Okay, 4x3 meaning there should be four rows and three columns. And but first of all, let me ask you guys, do you think this is even possible? What do you think needs to be true in order to reshape this properly? What do you think? If I want to take a a flat 1D array and put it into something that's 4x3, how many elements do I need to do to do that? Perfect. You guys are right on top of it. 12. Perfect. Yep, I need 12 elements. So, what happens if I don't have 12? Do we think this is going to work? Yeah, let's try it. Error. And look at what the error tells us. I cannot reshape something of size 11 into shape 4x3. It even tells us directly we can't do that. So yes, that is definitely a prerequisite to using reshape is that you need this total number of elements to match the number of elements that you start with and then it will work. So if you're going to use reshape, you can reshape into any shape that you want to. So, we could even reshape this into 3x4. That would be okay. We could do 3x4 because that totals up to to uh 12. We could do 2x six. That would be okay. But could we do 2x7? No. We don't know how to reshape something that is 14 into into a shape 12. But we only have 12 items. 2 by3 by two. Sure, we could do that. That'd be a three-dimensional. We could do that. And now we have a 3D array because we have uh each element is 3x two. We have two of them. So each matrix is 3x two and we have two of those. So we could do that. So that's what reshape does is reshape can take an existing array and move it into a new shape assuming that the shapes align properly. So reshape can do that for us. So that's actually incredibly useful. We'll use reshape from time to time. And then we can actually do the reverse of reshape. So we can always take something and flatten it out into a 1D array. Okay, we can always do that as well. So the flatten function can take something and put it into So this will always always give us a 1D array. So no matter what shape we start with, we can flatten it out into a one-dimensional version of it. Okay, by using the flatten function. So this this does a particular reshape that will completely flatten the array. So you can see it takes this uh three-dimensional array here and goes ahead and reshapes it or or basically flattens it right into this exactly flat uh onedimensional array. So flatten always returns a 1D array. Pretty pretty straightforward. Is there any benefits? Yeah. Sometimes we want to take something that is in one shape and move it into another because we're going to manipulate it. We we're going to assume it has a particular shape to manipulate it. We'll we'll see that later on when we get into deep learning. Especially when we work with images or text, it's going to be important to reshape things from time to time. So maybe not not maybe not the second, but when we get into deep learning, we'll we'll go ahead and reshape. Is that there is a transpose function. Now what this does is it swaps rows and columns. So it it transposes uh this into uh whatever was our rows. So this one two three now becomes the columns. And so this was a 2x3 matrix. It now transposes into a 3x two matrix. So transpose swaps our rows and columns. And that can that's going to be useful down the road too for different uh algebra calculations. We may need to do may need to transpose from time to time. Okay. So we're going to start with doing some arithmetic operations. Uh so just to show you guys that when you have data in numpy arrays, you can do elementwise operations. Meaning that we can do operations that go element by element, match them up and do uh some type of mathematical operation between them. So things like addition, subtraction, multiplication, division, we can do those uh between elements. Um, so for instance, we have these two arrays of the same size, the same shape, and we can go ahead and add them together. Meaning that like this uh position is going to be added to this position, this position is going to be added to this position, this position is going to be added to this position. Okay. So when we do that, we get um basically 40 in every slot because we get 30 + 10 is 40. 20 + 20 is 40. And 10 + 30 is also 40. But look at the syntax of it. There's actually two different ways to do it. You can do the numpy.add and then you pass in a and b. So this is the this is one way to do it. One way to add is to use mpadd and then you pass in your array one and array two. So you can do that. And so we we add those two and and store it in the result. And notice that the result is a same shape array but just with each element added together from the original arrays. So that's one way to do it. The other is you could just do regular uh arithmetic. So you could just do a plus b. This is an alternative. Alternative is to just use standard arithmetic arithmetic operations. It it doesn't really matter which one you do. I've seen both. Um both will result in the same kind of array. So we could store um we could do something like result equals a + b and then um store that in the result and then print um print the result. So it's same thing as before. Um we get the same array. So you can do either one. np.add a + b either one will do that elementwise addition uh between the elements. Okay straightforward. We also have the same thing for subtract, multiply, and divide. So for instance, when we have these, now we have a 2D array. So this is now two-dimensional. And but the same exact thing is going to happen. We're going to go through and subtract. This is going to do um this is this is the same as a minus b. So this takes a and subtracts b from it. So we have 30 minus 10 is 20. 40 - 20 is 20. 60 - 30. And then we do 50 minus 40. So we're subtracting those elements in the same positions to get a uh to get a result to get a result. This 2D array that is a result of subtracting every element from the original arrays. So again, you could do a minus b, you could do np.subtract, either way should work. Uh 2D plus 3D, could you do it? Uh, you could try it out. So, this is a 2D. We could try it out. So, let's copy this guy. Let's do Let's do this guy, which is going to be a Let's see. So, let's do uh a 3D array. So, let's do um let's do one of these guys. And then let's do another one, but let's just let's just change up the numbers. So, let's do 10, 15, 20, 25, 30, 45. Let's do np.subtract a and b. Let's see what we get. So we actually do get a result and uh the reason we do is something called broadcasting which is um an interesting idea in numpy that what they do is they basically will force the shapes to to be aligned when you do a mathematical operation but you might get some unintended consequences of doing that. For instance, we get like we get some of these uh actually work where we get 30 minus 30 40 60. So we get these zeros here for this first guy. But notice that we basically take this and apply it to this. It basically takes this and and subtracts to this guy secondarily. So we have 30 - 10 is 20 and then 30 - 15 is 15. Sorry. 40 - 15 is 25. And so the shapes, the dimensions don't have to be the same. Dimensions don't have to be the same, but you can get uh unintended uh results. So be be careful is the thing I would say. be careful of doing the subtraction of numpy will try to force the results to fit by taking this and applying it to this 2D matrix here. Right? So it it can work. You're just going to get um maybe some unintended results that don't really make sense but are possible in numpy. If you subtract a bigger value from smaller, it will no it'll give you a negative. Try doing in this example what would happen if we put B first you get negatives. So yeah it's still it's still possible you just get negatives there. Okay. So as well so we can take uh like 30 * 10 20 * 20 10 * 30 and we can get uh those multiplications. So if we do that we get this array which is going to be 300 400 300. So these are just elementwise multiplications. Again the alternative alternative is to do a * b um and that would be the same thing. So if we just did result equals a * b uh that would be the same thing but we can do mp.m multiply to make it more make it more uh explicit the operation we're doing that it's a time b and this is not to be confused with matrix multiplication. So uh that's something I should call out here is that matrix multiplication is a different matrix multiplication is different and will be covered later. So traditional matrix multiplication will be covered later in that uh that requires the matrices to be compatible and uh it's a completely different operation than doing elementwise multiplication. Okay, which is just a star b or asterisk b or mp.m multiply. We can do division. So um notice that this is again a scenario where the shapes are not the same. We have a 2D array and we're dividing it by a 1D array. Now what happens is we basically take this and divide it by this and then take this and divide it by this and get this second row. So this size is going to basically match the larger of the two shapes. So this first shape is two-dimensional 2D. So basically the shape of this is um 2x3 shape and this is only a basically a a 1x3 not even it's just a three element shape it because it's just a 1D array. So it's a 1D array. So therefore, therefore um when we do the division, it's going to do that broadcasting thing that I mentioned earlier and try to force this to be able to divide by this. And the way numpy will do that is say, okay, which is the bigger shape? This is the bigger shape because it has more dimensions. Try to take this and divide it by these guys. What do you think's going to happen if we reduce this size? If we did this, do you think this would work? If I reduce this down to only having two elements, do you think this division would work? Error. It's It doesn't broadcast that way. No. So, you could even see it. It even says it tries to broadcast, but it doesn't know how to. Um, so the even even with broadcasting it the shapes still need to align to some degree. So, this still needs to be like um so needs to be a valid shape to be able to broadcast to each one of these uh dimensions here. Let's try it. Yes, we can divide B by A. We're just going to take now this again is going to try to match the shape and just do this divided by this. this divided by this. But remember that a divided by b is not the same as b / a. Not the same result. Yeah, you can get a division by so uh try making one of these zeros. So try let's say this was all zeros and we did a divided by b. Do you think that'll work? Do you think this will work? Yeah, it basically now it it technically it technically returns a result, but they're all infinity. Yeah, it basically says that we have a warning. We cannot divide by zero. So, it basically says that uh sure you could tech you could do it, but you're going to get infinities all over the place and it's not you get a warning that you're dividing by zero. So it's kind of like an error there. Okay. All right. So let's go to doing exponents. So we can do uh elementwise exponents where every element in A is raised to a power of something in B. So for instance we like two squared we do 2 cub 2 4th 2 to the 5th 2 to the 6. And you could see what each one of those would be is this. So this just does um every element in a each element in a gets raised to the exponent of the corresponding element in B. So we get 2 ^2 as I said 2 cub 2 4th 2 5th 2 6 that may be useful from time to time we may have a reason to take elements from one list and expo exponentiate them from another list um so there's a power function here to do that so that may be useful uh before we move on to the statistics functions which are going to be really interesting. So in fact I can just show you a quick example of that. So we could do um we could do we could call a numpy array from an existing. So we could have inside of a a list we could have mparray and then we could have one two three and then we could have um an MP array and then we could have um four five six. So this should make so if we then we uh let me display x. So this this makes a 2D array. Does this make sense? Like I'm I'm defining an array as the input elements to build an array. >> Okay. So what's great about numpy is with the arrays it can easily do statistics on arrays. So it can find the medians, it can find the average, it can find the standard deviation, it can find the variance. Now I know we haven't uh technically defined what each of those are, but that's okay. It's useful to know that given the different statistical functions that we may want to do, numpy can easily compute them on collections of data, collections of arrays, right? Or data that's inside of an array. So for instance, if we have this 2D array here, we can compute the median of all elements in the array, which would be npmedian. So there's a builtin function from numpy npmedian. Um so we can do npmedian um finds the median of an array. Okay, so this will calculate the median. So if you're uh again we haven't we will get to what the median is later on in our statistics overview later on but the median is kind of the 50th uh percentile like the middle element right the middle element of a we we basically order it from least to greatest and find that middle element um so in this case the middle element is four out of all these elements that we have okay npmedian uh we can take the average which is the mean. So that's very nice. We can take the average um so we can do np mean which will take the average of this array. So it'll average all the elements uh together and we get an average of 6.3333. Um so that's very convenient for us that we can compute an average of an array. I want you to now this seems really straightforward but I want you to see how powerful this is. is that lists for example do not have this ability. There is no builtin mean function for a list. Um so that's why this is so useful that numpy has this ability and it's really optimized. It's really fast to find the mean, really fast to find the median. It's a really optimized function to do it. If we wanted to find the average, we would have to do it manually on a list. We'd have to total up all the elements and divide by the size of the list. Um, not that that's hard to do, but it is something manual that we would have to define. It does not automatically exist like what we see here with MP. As a simple function that we can apply and uh we can apply it to numpy arrays very easily to compute the average. So, same thing with standard deviation and variance. Those are uh more advanced statistical functions that um figure out the spread of the data away from the average. Uh again, we're going to learn about these later on, but MP. STD does the standard deviation and then var does the variance. So, and and really the standard deviation is the square root of the variance. So, if you to if you took this um variance, sorry, if you took the standard deviation and just uh raised it to the second power, you would get the variance. But uh you can compute them separately this way. Okay, so pretty convenient that NumPy provides those statistical operations. We're going to be using these quite a bit when we especially when we do like uh exploration of our data and we want to find an average. We want to find how what the standard deviation is. Um this is going to be really useful to use the numpy function to compute that on a on an array of data. Okay, really useful. Okay, let's talk about uh percentile then. So, numpy also has a percentile function. So, we can take an array and compute the 50th percentile which would be the uh median. Again, we haven't learned what a percentile is, but if you think about uh ordering all the elements and figuring out like the median is at the 50th percentile and then um half the data is below that. So there's a point where like 25% of the data uh is below this certain value and then 75% of the data is below this certain value or um 95% of the data is below this value. Um so the percentile is something that ranges between 0 to 100 we can take. So like the 99th percentile means that most like 99% of the data is below this certain value. Okay. So like 99th percentile means that 99% of the data falls below that value if we ordered it and kind of sorted it that way. So we can compute any percentile we want by just passing in the array and then giving it a a number between 0 to to 100. Okay. So and this should be a whole number 0 to 100. So for instance we can compute the 99 percentile. Oops. I have to actually run this. Let me run that. There we go. So the So 99% of the values are below 22.8. Um, which kind of makes sense because most numbers are pretty low. So there's really only one number below that number, which which is the 24. So if I did if I did the 100th percentile, that would basically be the max, right? basically be that 24. So only only 100% of the numbers are below the max. So it kind of makes sense. And then the if I did the 50th percentile that would be the median. Half the values are below that half are above. So that that matches the uh median that we found here which was four. Okay. Then we can do percentile. All right. Finally uh wanted to mention that numpy you can manipulate strings in numpy. Now, this is uh less often used because typically when we're working with numpy data, we typically don't have strings inside of there. We usually are dealing with numerical data, hence why it's called numerical python numpy. But it is possible to work with strings and do different string manipulations. So for instance, if we have a numpy array that has two strings, hello world, and then another array that has welcome learners. So these are two 1D arrays. We can actually concatenate elementwise strings by using the MP uh character module. And instead of doing the the the reason it's inside of the character module is instead of doing like a numerical addition, so typical arithmetic, it's doing a string addition, which is concatenation. So it's doing character addition character addition here which will um concatenate these two strings hello and welcome. So those end up merged together concatenated together and then world learners uh get concatenated together. So this is the uh string concatenation uh elementwise string concatenation. So it's from the MP char or character module within numpy. Okay, very rare. It's very rare we would have to do this, but I'm just pointing us out that it does exist. Okay, if if for some reason we need to manipulate strings, uh we will have that ability to. Okay. All right. So then we can replace uh substrings with new strings. So uh if we have this original string called hello, how are you? Um we can print it out and we can replace uh we can use a character replacement to replace within this string replace hello with hi. So once we do that we can uh print out the new string. So um then we get hi how are you as the new as a new uh string. So this does a um string replacement um if we can find the uh substring. So if hello exists. So uh we should test this out and see like uh is there something so we could just put in something that doesn't exist in there. You know it's not going to be replaced. So if we this is saying okay let's let's try to do a replacement of this string. Let's replace something with high. But something doesn't exist. So, it's just going to return to us. It's not going to replace anything. It's just going to return to us the original uh string. But this does exist. So, that's going to be replaced with high and do a string replacement. Um, if we want to, we can also uh manipulate strings. So, do uppercase everything, lowercase everything. You can see how those uh like this is all lowercase but we do upupper pass in the string it will uppercase everything. Um this is the this is all uppercase we can lowerase everything. Um you can see how that all works. Okay again very rare we would ever need to manipulate strings but if we did there's this character module that can uh help us manipulate strings. very rare that we would need to because most of our data is going to be numerical inside of a numpy array. All right, so just to recap, we have our arithmetic operations. Basic arithmetic between numpy arrays we can do pretty straightforward. Um we can we can even use our standard we can use MP add or we can use a plus b, np subtract, a minus b, multiply, divide. We can all use those basic um arithmetic operations. We also have a power function which will raise things to exponents. These incredibly useful. We'll use these all the time going forward are the statistical functions like average, standard deviation, variance, median. We'll be able to compute really easily on an array. Okay. All right. So let's go to our next notebook and continue working with numpy. So we go to the 3.04 notebook. All right. Now the whole point of this one uh is to practice accessing data within the notebook or sorry within the numpy array. And truly this is going to be great because it's going to work exactly the same as a list. We're going to be able to access elements by their position and also slice just as we did with lists, right? So everything's going to work the same, which is going to be really nice. The one unique difference is that with numpy arrays, we can have multiple dimensions. So that's where we need to actually be careful is that if we want to access elements that are in different shapes like they're in a second row, third column position, how do we do that? And it's actually going to be really easy to do um if we just think about it as kind of a coordinate passing in like an index as if it was a coordinate of this is exactly what I what I want to access. Okay. All right. So, if you take a look at this picture, this is a really great picture to break down a 2D numpy array. So, imagine we had a 2D numpy array whose shape was two rows by three columns. So, it's a it's a 2x3 shape. All right? And these we have elements 1 2 3 and four five six in our array. So, we have two rows. Each row has three elements. Okay? Okay, so a 2x3 shape. Now the element that is right here is at index zero because it is at the first row. So it's at index zero in terms of the row, right? So so there's two rows. So it's either going to be index zero or index one for the row coordinate. So it's at it's at index zero for the row, but which column is it in? It's in the first column. So it is at index zero for the column. So this coordinate for this guy would be like if we passed in 0 comma 0 as the index, right? So if we passed in 0 comm, 0 as our index, we could access that element right there because what this is signaling is we are at row zero and we are at column zero. column zero. Okay. So with multi-dimensional arrays we have to be careful about that is that things can be accessed by their coordinate now their their index coordinate rather than just a single index like we saw with list right with a list it was just okay we can grab something at index zero index one index two maybe index minus one but with a 2D array we're actually grabbing things uh at their coordinate right so row zero column zero is This should return this should give me the element one which is this guy right if I were to access that element. All right so let's try accessing. So let's take a look at this guy. This guy is now going to be at same row. So this guy is still going to be at row zero but it's going now be at column one. Right? So it's now it's now here column one. So we should be able to access that two sitting at row 0 column one index. So this should this should equal two. Right? This item should be that element two. Okay. And then lastly from this same example we have this three. This three should be we should be able to get from we're still within row zero, but we're now at column index two. And this should equal this should equal three. Okay, so in a 2D array, this first entry is what row we want to go to. So imagine like scanning over this grid. What row do we want to go to? Okay, row zero, that's the first row. What column do we want to go to? Okay, column two, that's the last column. That's the third column here. Let me ask you guys, does that make sense in terms of thinking about it like a grid and a coordinate of how we access elements? Okay, so let's see how this works inside of code. So if we create a let's create some arrays. All right, so let's create some some arrays to practice with. So, so we create a 1D array, we create a 2D array, and we're going to create a 3D array so we can practice accessing certain elements. So, if we look at the 1D array, this behaves exactly like a list. In fact, I'm going to write that down. This behaves just like a list. the 1D array we can access like the third element or the first element or you know in this case we're accessing the fourth element which is at index three um and that returns to us the four. Um so we're we're doing that just like a list. In fact we can also do the very last element um which would be at the minus one index and that is the six. So it it behaves a 1D array behaves exactly like a list. Not much different in terms of accessing the the elements there. We don't need to worry about coordinates because there's no rows and columns. It's just a single. It's basically like a list, right? Very easy to access things. Okay. So and then and for instance, we can add two elements from these positions together. This is this adds the position one and position zero elements together. And so that ends up being three. And we can see that because that is uh 2 + 1 which is uh three. So pretty easy to do. All right. So here is that exact uh here's another picture of everything we just drew earlier where we're thinking of the elements at these positions as coordinates. Right? So this the element right here in the first row first column is at coordinate 0 0 and then the element right here is at coordinate 01 02 and then if we go down to the next row this element would be at row one column zero index row one column one index row one column two on and on and on you know however many rows and columns we have. So we think of accessing elements that way by their row and column index in a 2D array. All right. So for example, let's get the element that's in the first row. So this should be the first row because that is index zero and this should be the third column. This should be the third column, right? Is the index two. So if we go back to our 2D array, it should be the first row, which is this guy, and then the third column, it should be a three. And it is, right? So if we print that out, we get a three. How do we feel about that example? Do we see how we're accessing it from inside these brackets just like we would a list, but now it's a coordinate. Do you see that one? Okay. And uh we also have now we can grab something from the second row and from the uh from the second row and this because this is now one and this is now the second column. So we go back to our array second row second column should be this guy here right second row is this guy second column is this guy. So this should be the five, which it is. We print it, print that out, we get the five. Okay, that's what that coordinate represents. Second row, second column. All right, very good. Um, I want to take a look at a 3D array now, which is going to be a little interesting in that we're just going to add an extra coordinate to the mix. So with the 3D array, we basically need to know which matrix are we talking about. So remember, a 3D array is going to look like this. We basically have rows and columns, row, column, row, column, row, column. We basically have an array of matrices as our 3D array. So the first coordinate is going to say, which matrix are we at? Are we at this one? Are we at this one? Are we at this one? Okay. All right. So, in the in the 3D example, what I wanted us to see though is that notice notice that like the first coordinate is going to say, are we talking about this matrix or are we talking about this matrix? So, we're actually going to have three coordinates. So, this first coordinate is relating to which matrix are we talking about? And then once we know what this is, like this zero would say, okay, we're talking about this first matrix. Then it's the same as usual with the these two coordinates are these two coordinates here are now what row and column within that matrix are we talking about? Okay, so with 3D that first coordinate is which matrix is it? Is it the first, the second, the third, the fourth? Because remember in a 3D array every element is a matrix. Every element is a 2D array. So the first coordinate is saying which matrix are we talking about. All right. And then once we know which matrix it is, we can use the next two coordinates to figure out what row and column of that matrix are we talking about. So let's see an example. So look at how this 3D has three elements. So if we if we break this down, this is saying that we want to access within the second matrix. That's what this first coordinate is saying. Within the second matrix, I want to I want to access the element that's at row 0, column zero. Right? This is the second matrix. Second matrix. So in a in the setup it's like we have a matrix here, we have a matrix here, we have a matrix actually we only have two in our example. So this element means we want to look at that second matrix and we want to grab the element that's at coordinate 0 uh coordinate 0 comma 0. So let's let's go back to our matrix and see what that should be. So what is the second matrix? It's this guy. This is the second matrix that is within our array. And then we have we want to access row 0 column 0. That should be this. It should be this seven. Right? So row 0 column 0 of that second matrix should be that seven. And that's exactly what we get. If we print out this this coordinate, we get seven. matrix refers to row. Uh kind of. Yeah, it's because in a 3D in a 3D array, every element is a matrix is a 2D matrix, right? Okay. All right. How do we feel about this three three-dimensional indexing? Does that make sense? This is matrix matrix that this is the second matrix and then this is row 0, column zero of that second matrix. Finally, I want to mention that we could do negative indexing for all of this, right? So that still applies. Like if we did minus3, that would be, you know, third from the last. So if we go back to our 1D array, we could do minus one, we could do minus3, minus1, minus 2, minus3 should be the four here. Um, so we can still use our negative indexing as we would with any list. So that does not change. we can still use minus indexing. Um we can even use it in the 2D array. So if you imagine there this is now in terms of the column because now this is saying in a 2D array we want to grab the element that's in the second row but in the last column. That's what the minus one would mean in this case. Second row but the last column because we have a minus one index there. So, second row, last column. Let's go verify that's a six. So, second row, second row, last column would be this guy. Yep, that would be a six, right? Because this is this is our second row and this is our last column is a six. So, that's that's a uh that's a nice thing. That's a convenient thing. We don't need to know exactly how many rows are. If we just want to grab the last one, just a minus one. So this is uh second row last column and then if you look at the 3D example we have in our second matrix the second row last column again. So we can grab that. So that should be second matrix but then the second row and last column. So second matrix would be this guy. Second row is here and then last column would be the 12. Right? So there it is. There's that coordinate. Gets us to the 12. All right. Okay. The natural next thing to do is to take a look at slicing. Uh because we know we can index into a multi-dimensional array. We should be able to slice those as well to extract multiple elements at once. That would be the next natural thing to do. So, we're just going to extend everything we know from list slicing. We're just going to apply it here. It has the same exact syntax as what we did with list where we do start and end or start in step like we just like we did with list. And so all those rules apply. It's just now again we have multiple coordinates to to consider here. So we can slice uh rows, we can slice columns, we can slice uh in 3D like matrices, rows and columns. So uh we have all of those options at our disposal to do slicing. So when it comes to a 1D array though, that's going to behave just like a list. So, uh, when we run this, um, this should slice just like a list would where, for instance, we're going to slice starting at one going up to seven, but not actually including seven. So, 1D array behaves just like a a list behaves just like that. No surprises there with the slicing. It's we've already learned that before. We can slice just like with that with the colon there to slice out a section of that list or of that 1D array I should say. All right. And so uh same thing with like having uh a starting point but no stopping point. Remember what that means is we start here and go all the way to the end of the list. So no stopping index means we slice to the end of the of the array. In this in this case it's an array. So we start at index five which would be this string and and include everything to the end. Right? So it should be those um we can have a step value. So this would be every third element uh starting at index one. So this should be uh starting at seven and including every third element. So that would be four. And then there's really no third element beyond that. Um so it just would be seven and four. Okay. So we can have a step size there. We could do every other element, right? Which would be slightly different. So every other element will be seven, five and three. So, that's all pretty standard. Um, nothing that interesting going on there. I think what's going to be more interesting is slicing in the 2D array setup. So, if we do that, let's imagine we have a 2D array. Now, what can you guys tell me? What is the shape of this? What is the shape? What do you guys think the shape of this array is? The shape, not the not the number of elements, but the shape. 2x five. Yep. It's remember the shape is number of rows and columns. Number of rows and columns. So it's two there are only two rows because there are only two lists and uh and we have five entries in every list. So it's a 2x5. So it's a it's a 2x5 that looks like this. 2x5 where we have 11 22 33 44 55 and then we have 66 77 88 99 110. So we have a 2x5 matrix that looks like that. Okay. Now let's go ahead and slice. So what I want you to pay attention to is notice how we are allowed to slice in each dimension that we want to. So for instance here this means let's take the first row that's what that's what this means first row but let's grab the se index two column which is the third column and go up to the uh the index three column but we would not actually include that. So it would only be this index two column. So that should just be row zero and it should just be this guy here. This is the index two column. We technically would go up to this one but not include this one. So we really only get 33 from that slice. So let me write that down. So here we select the first row and slice the columns. So we can do that. We can actually do any combination of that that we want. We can slice rows, we can slice columns, we can even do um so let me show you another example here is that we can do we can uh swap this to do we can grab the first we can grab the first two rows first two rows essentially. Um remember it's not going to we basically are going to go all the way up to the second row but there there's only uh index two row but there is no index two row so it's only going to it's going to include it should include both and then we can include let's do the let's do the first column index one column which actually be the second column up to the third. So that actually will grab that array right there. So, this grabs this 2D matrix right here. Now, why does that happen? It's because we're including both rows because we're slicing out zero, row zero, and row one, right? So, we get both of those rows, and we are slicing out the first uh the second column, which starts here, and we're going up to the third column, but not including it. So, we should get these two guys here, which we do. Okay. So, if I were to draw this out, I think it'll be clear when I draw it out. So, let's draw out the matrix again. We have an 11. We have a 22. We have a 33. We have a 44. We have a 55. And then we have 66, we have 77, we have 88, we have 99, and we have 110. So what we are slicing out is we're slicing out the first two rows. So, we should be looking here within the first two rows, but we're only grabbing uh columns that start at index one and go up to index three, but do not include index three. So, they should be sliced out to really here all the way up to here. Okay, does that make sense on that slice? 0 to two, 1 to three. Notice how we're slicing in each dimension. We're grabbing a certain collection of rows and a certain collection of columns, right? We're grabbing a certain collection of rows, certain collection of columns. Yeah. So, uh how we got 33 previously is that we um let me clear this out and then let me um undo this. So, it was this index um that got us there, which was to say we're grabbing something from the first row. That's index zero. So, we're looking within this row, and we're we're going to the second column up to the third, but not including the third, which is what gets us the 33. Because this the second column index is starting right here. we would go up to here but we don't actually include that. So we only get the 33. We don't count the first index to start for which operation. Uh no we don't we don't count the we don't count the last index. So we don't count like this one does not get included. The the last index does not get included on the right side of the colon for the stopping index. It doesn't get included. Up to but not included. Right. Up to but not included. Okay. All right. So, we can slice a 3D array as well. So, if we take a look at this, we have again um a situation where we have, let me draw it out again, where we have uh two matrices. So, we have um a 2x3 uh and we have two of them. So, we have inside of here we have one, two, three, and then we have four, five, six. And that's our first matricy. and then or matrix and then we have um 7 8 9 and then 10 11 12 as our second matrix. Okay, so that's what our 3D array looks like. And what we're saying is let's go ahead and take the first matrix. So if we were to build out the slice, we should be taking the first matrix, which would be this guy. That's what this that's what this tells us to do, right? This zero tells us to take that first matrix and then let's slice the first row. That's what this means. So go to the first row or sorry index one row which would be this index one row and then uh every column past the first. So every column past the first index to the end. So we should be slicing out these two guys. which is what we get. We get five and six. Okay, so that's what that So again to recap that this tells us to take the first matrix. This tells us to take the second row and anything past that second row. So here we only have two rows. So it's we're only looking at that second row. But if we had more, it would be all rows past the second row. And then uh then we're looking at the second column and anything past the second column. So in this case, five and six are in those two columns uh that are past index one. Five and six. Okay. How would you do up to a certain row? You would um basically reverse the colon here. So you would do like if you wanted to go up to index one, you would just put like you would do this, right? That goes every row up to that one. Does that make sense? Up to but not including that. If you want to include that one, then you bump this up to two, right? So let me make a comment here. This does first matrix each row pass index one and then each column pass index one. Okay. So this example we had was a 3D matrix that was like this, right? So it had it had two um 2x3's inside of it, right? So it had 1 2 3 and then four five 6 and then it had a second matrix that was 7 8 9 and then 10 11 12 right so this was our this is the this 3D array that we have okay so with the uh 3D slicing um what we're saying is we have three coordinates we could possibly slice, right? Three coordinates we could slice. This first coordinate says let's just grab the first matrix. So the index zero matrix which is uh which is this guy here. So let's just let's just zoom into this guy. That's what this says. Do we do we agree on that? That's what the index zero there means is let's look at the first matrix. Okay, perfect. Okay. So then within that first matrix, we're going to slice according to these two, right? So this first one says let's look at the index one row and do everything beyond that to the end. Now the index one row is let me draw it out in green is this guy. Do we agree on that? That's the index one row. And and if we were to go to the end, that's the only that's the last row. So in theory, if there were more rows, we would keep going. But that's the last one. That's what that colon says is go all the way to the end. Do you agree? That's the index one row. Four, five, six. Does that make sense? And we're slicing. The the colon means slice to the end, but that is the last row. So that's that's the only one we would be using. Okay, now let's look at this last one. This guy says we should be slicing column index one. Column index one and then to the end to the end. Now what is column index one? Column index one starts here and goes to the end. So if we were to slice out what we have in this four, five, six row, where is column index one? That starts at five and it should go all the way to six. So we should just end up with 56 as the result, which is what we do. We get a five six as a result. We get that one row. We get that one row and we get the five and six out of that that row because those are the that's index one to the end of the columns. We could test it out. So let's try let's just insert a zero to two here and see what we get. Yeah. So 5 6 and 112 because it's going to it's going to do that to both entries. It's going to go to both rows or sorry both matrices. So to this one and to this one we're going to slice uh we're going to slice out this and this. Okay. Okay. So don't forget we can use uh negative one inside of slice. So for instance uh what this would say is let's do everything up until the last element. That would be this example here. Just a simple example to show you that we could do everything up until the last which would be minus one. And of course we could use that in a 2D example with like everything but the last column, everything but the last row. Uh we could always slice that way. All right. So we can't forget about negative slices or negative indices. We can always use those whenever we slice. All right. So, just to kind of give us a a short little quiz here to see if we're uh following along. Um, let me ask you guys, what is Numpy and what is it used for? What do you think is the best choice here? Very good. A lot of votes for B. B is definitely the right answer. None of the others really make sense uh for what we're talking about. B is is a great choice. Perfect. Uh, good job on that. NumPy is used uh to do to to manipulate data essentially in these science and engineering applications especially data science. Okay. What are some of the key attributes of the ND array? What are all the ones that we kind of covered? This this one might be tricky but if you remember them great. Which ones did we cover? This is mixed votes for A and B. Let's see what it is. Should be A. So the reason it's a is because there's really no there's really no index attribute. We we can index into the array by using the um brackets to index into it and access elements of the array, but there's really no index attribute that's useful to us. It's more about the dimension, shape, size, data type, item size. Um those are definitely important. So A would be the best choice here. That one's that that question is a little tricky. All right. Which function do you think is used to change the dimension which is used to change the dimension of the array? Awesome. You guys are pretty unanimously voting for a yes. Reshape will reshape it into whatever dimensions we want to. Uh provided that there's enough elements to do that, right? Provided there's enough elements, we can reshape it into that shape. So that that's great. Good job. All right. Okay. So, let's talk about pandas. Remember, pandas is an open- source library built on top of numpy and is used for manipulating data. So, it is going to be the thing that we're primarily going to use with our uh kind of spreadsheet table like data that we will be working with throughout the throughout the um course. And it provides so like numpy has the numpy array as the primary data structure that we manipulated previously. Pandas has two really important structures. Basically it has uh a what's called a series which is going to be essentially a a fancy array. So it has a series and then it has a data frame which is like a fancy matrix. So it's like a collection of series. We're going to talk about both of those and do examples with both of those. But those are the two data types that really make working with structured data efficient. If we could put our data into a data frame, for example, it's almost like having a spreadsheet in Excel that we can do a bunch of operations on. So dataf frame is kind of like a table. It's kind of like a table or or a sheet in in like Google Sheets or or Excel spreadsheet. That's what a data frame is kind of like. So that's going to be extremely useful for manipulating data, right? Like grouping it, filtering it, uh doing operations on it and eventually modeling with it. Um we'll want to have things inside of a data frame. A data frame will be something we'll use a lot going forward. A series is kind of like a a column within a so it's basically like a column within a table within a table um or within a sheet. So it's it's like a single column within there. So a data frame is actually nothing more than just a collection of series because a table is just a bunch of columns uh together right inside of inside of that sheet. So we'll first take a look when we go through pandas we'll first take a look at working with series because we need to know how to manipulate individual columns and then we'll look at the data frame as a whole uh which again is going to be that collection of uh collection of series uh collection of columns. So the series and the data frame are going to be our two main data structures that we'll work with. Again, the reason we work with pandas is it provides us really nice data frame and series objects. It can do a lot of operations on data um to help us clean up. It has a lot of really nice like cleaning functions, a lot of transformation functions, a lot of summarization like it has group by capabilities which is really nice. Um it has a lot of loading and and exporting functionalities which are really nice. So, pandas is really awesome. It's a universally used tool in the industry. People work with pandas every day. I work with pandas all the time. A lot of people I know work with pandas all the time. Um to manipulate data. Okay. So it's a really really important uh package in the data science uh ecosystem. Um so it has really nice uh data management and and transformations. That's kind of what wrangling means. Has really great aggregations and transformations. has really awesome tools for reading and writing. As I said, like loading in from a file. That'll be a pretty typical operation for us is we'll have like a uh a CSV or an Excel sheet and we'll actually load that into Panda so we can do manipulations with it. Um it has joining and merging uh capabilities almost like a a a table in a in a relational database. So really nice operations there. It has really nice indexing capability and alignment. Um, so we'll see that uh as we go along with the series and the data frame. So, a lot of nice features, a lot of really nice features that make working with data super convenient and easy. Pandis is really nice. So, as I said, it has those two structures. is it has the series which is kind of a one-dimensional uh uh array but the series is like a fancy numpy array in that it actually has a label with every entry. So instead of just having a positional index which every array has right we can access things by their index a series actually has a label which is really nice. So think of a series kind of like a small table where we have um basically something like this. We have a label and then we have our data and then we have maybe one, two, three, four, five, whatever. And then this label can be whatever it is. It can be um it could be a string, it could be a number. So the label could just be a simple index like 0, one, two, three, four, on and on and on. It could be something more fancy than that. It could be like a string label like um category 1 um category 2 and on and on and on. So a series provides us the ability to have an array but with an additional index with like a label index that we can use if we need to. So it's a series is really nice in that way. uh but it is one-dimensional meaning it only has one array one array and most of the data we work with is going to be in two dimensions so it's actually going to be with a data frame. So the data frame is a collection of series. So you think of a data frame like this where it's a series a series a series a series and series every column each column each column is a series. But what's nice about a dataf frame is even though each column is a series we basically have a unified row label. we have a row label that we will uh discover is going to be really nice for uh maybe we want to access uh certain rows by their label and not just by their index. So maybe some rows are assigned like a string label, a date label, you know, whatever whatever we want it to be. Okay. So, our job over the next um like our next session, unfortunately, we're basically out of time for today, but our job over the next lesson will be to dive deeper into these two structures. So, we're going to learn more about the series and how to work with series, how to manipulate series, load data into a series, all the important functions that come with the series we're going to learn about. and then we'll graduate that information into the two-dimensional version of that which is going to be the data frame. Okay. So let's that being said let's start with the series and then we'll work our way up to extending that into two dimensions. So we're going to start uh by looking at the series. So in general as I said a series is going to have it's going to have data which is going to be that one-dimensional array of data and it's going to have an index. So and again that index can be fully customizable. It can be by default um just the standard positional index. So just like we would see in a list or in a numpy array it would be 0 1 2 3 just like you see in this little picture right here. This would be kind of the standard like positional index of of the series. And then this is your this is your data here. um that is the underlying data of the series. What's really nice about pandas is it actually builds on top of numpy. So this underlying data is actually stored by pandas as a numpy array. And so because of that um series kind of inherit all the nice properties of numpy arrays in the sense that they can do all the same arithmetic. They can be manipulated in a lot of the same ways. they have a lot of the same attributes like a shape um and a reshape function and all and a transpose and all these things. So, so that's really nice about really series and data frames. The underlying data is all built on top of numpy um which is nice. So you can make now you can create a series um out of a list. You can create it out of a numpy array. You can make a series out of um a tupole. You can make a series out of a lot of different data. It's just that one-dimensional array that has that special kind of index to it that we could we could customize. So we're going to see different examples of building a series and manipulating a series. The other thing I should mention is that there's no restriction to the kind of data that the series can hold. So it can hold objects, it can hold strings, it can hold integers, floats, really all of those. And it can even be mixed. So just like just like in a list, um we would see you know we can have strings, we can have floats, we can have those kind of things. All right. So let me show you an example. And the first place we'll start is by importing pandas. And this is just like we did with numpy. Remember all of our numpy examples had import numpy as mp. All of our examples here with pandis are going to be import pandas as pd. This is something we're going to get super used to as we go along is is the fact that we see pd. That instantly should tell us we're using pandas. That's the industry standard. So industry standard alias for pandas is pd. So if you're reading some code and you come across a PD that is just short for uh pandas and that that's the standard in the industry. So everyone would understand you if you were using PD there. So we import pandas pd. Look how easy it is to make a series. We just start with a list. So we just have a regular list of 1 2 3 4 5 just those those five numbers. And we create a series by doing pd. and we pass in that data as an argument to the pd.series function. This constructs a series and we store this as a series uh object. So what's interesting about this is when you first create the series, if you don't specify an index, so if you don't specify the index, pandas uses the default positional index. So it just assumes that the data is going to be uh indexed by position. So that's just, you know, this one would be at the zero index. This two would be at the one index. This three would be at the two index and on and on. So it's just the standard just like we learned how to slice. Just learn how to access elements by their index. That's all that's all we would need to do to access these items of the series. So that's if you don't specify an index. But I want you to see this example here where we use the same data but now we actually customize the index to be you know special labels for every item. So for instance, a sorry uh one would be indexed with a two would be indexed with b three would be indexed with c and and so on. So the way we do that is when we create the series, we actually explicitly pass in an index array which signals that these elements are going to all all this data is going to be indexed by this specific index. And then why that's relevant is because when you print out this series, you can access things. Look at how down here we actually access things according to this B index. Or obviously we could pass in a C or a D or an E and that's how we would access things. Okay. But in the in the uh series that does not have uh that just has the default index notice that we can indu just positional index. So we could access the first thing could access the second thing and then this would access the third thing which is at index two. Okay. So and and by the way you can combine these. So you might be wondering, well, we we have some data and we have an index. Why don't we just map those two together? And we can. So if we use a dictionary, we can actually combine those two together and say, okay, A should map to one, B should map to two, C should map to three, D should map to four, and E should map to five. And we can create a series from this dictionary. So you can actually build a series from a dictionary. And what this will do is it will treat all the keys as the index uh values and it will treat all the things that they map to the values right will be the data that'll just be the data right all of these 1 2 3 4 5 will be the data and this E D C B A will be the index okay so when you do this when you use a dictionary keys become the index and values become the data We can use a dictionary. We can just use a regular list. You can also use a numpy array. It doesn't have to be a list. You can use a a regular numpy array to to create a series. So all the code here is just creating a series. Nothing that interesting going on so far. Just creating a series. And I want you to see if we were to print out I'm going to add this in here. Just one little cell below this. I'm going to print out the series so you can see it. Let's print the series. You can see that it it's going to have notice how the series prints out with kind of two columns. This is indicating that this is the index 0 1 2 3 4, right? It's the positional index of this regular series. And then here's the data, the 1 2 3 4 5. Okay, so you you see both of those. Let's print out this series um with index. And let's print that guy out. And you can see now that the index is A B CDE E. That should make sense. This is kind of like an Excel column, isn't it? Because Excel, if you're familiar with those spreadsheets or even in Google Sheets, right, every row has this index A, B, C. Like you're you're dealing with cells that are in row A, B, C, D, and on and on, right? Let's take a look at uh series with dictionary. This should be this should be series from dictionary and we get the same thing the abcde e 1 2 3 4 5. What is the difference between dictionary series? Well, a series is a different object. A a series is a a pandas object and therefore it has certain functions available to it like finding an average um plotting finding a a mean or sorry finding a median like it has a lot of built-in functionality as a series that you don't get as a plain dictionary. Does that make sense? Like you get a lot of we're going to see this in pandas. You get a lot of functionality if you're a series. You get a bunch of statistical functions that can be easily ran. you get you can do a bunch of summarization on on that data. Um you can do plotting against that data relatively quickly as a series versus as a plain dictionary. It would be it would be object. Try it out. So try it out if you so if we did um so let's say we did a string inside of here. So let's make that three a string. So we have integers and strings. Let's now print this out. They basically, do you see how they're all objects? Which means that pandas would actually convert everything to a string. Once it sees one string, it's going to treat all of these as a string. So if you see a data type as object, that that likely means everything is a string because string is technically an object. So we could convert that back to convert that back to an integer. rerun that goes back to an integer. Okay. All right. Let's see what we get. So, I think it's a valid question. What's the difference between a dictionary and a series? Because it it's getting at like what's what do we get by having a series that we wouldn't have if we just had the dictionary? And it turns out that we get a lot of different functionality out of the series that we wouldn't otherwise have. So, we're going to see some of those functions. The first couple of functions I want you to to take a look at are right here, which are the head and tail function. So, what these do is they give you um a a view or essentially a copy of the first n rows. So, whatever in whatever value you put in here, you'll get those first few entries. Head goes from the top. So it goes from the beginning of the series um and does the first however many rows. So if you don't put a value in there and you just have this, the default is five. The default is the first five entries. Okay? If you don't put anything in there, you could now you could put a value in there like 10 and you'll see the first 10, right? Or you you want to see the first three, you put in a three. So um head gives you that copy of the first three elements and um hail gives you on the reverse end it gives you kind of that slice. Think of it as like a shorthand for a slice. It it gives you a slice of those last five guys. So this is almost like doing this is similar to doing series and then we were to slice out the like last five entries. Think of it like doing something like that. And then this is similar to this is similar similar to series and then we do the first first five like that. Okay. So head and tail very uh very interesting uh functions that actually we will use quite a bit with data frames um particularly to just get a quick view of the first first five rows and last five last five rows with a tail. So if we were to run this we could print out we could print out that. So, we could print out the first n rows and we get the first five entries, right? Um, we could actually do the first two and that would just give us the top two, right? From from the top. Now, we could print out the uh last last two, let's say. So friends, last and rows and this does the last two, right? All right. So head and tail pretty straightforward. They give you the first either from the beginning first two or from the end the last two. Series have a shape to them which makes sense because the underlying data is a numpy array which also has a shape. Now let me show you what the shape is. Let's print the dimensions. print that and let me comment that out. So, as you can see, this series only has five elements and it's basically onedimensional. So, even though it has an index, it is one-dimensional. It doesn't have rows and columns. It basically only has rows as five rows of data. Uh it's it's just a single column essentially, right? It doesn't have multiple columns. So we just say it has a shape of five. It's just onedimensional data, right? So a series a series is always 1D on one dimensional. Okay, it's always onedimensional. All right, now let me show you a really awesome function called describe. Now this is a cool function to uh give you a bunch of the statistics basically give you a summary a descriptive summary of your data. So when you have this is the advantage to having your data in a series is you can easily run a describe function and see how it's series.cribe. What this is going to do is give us a quick descriptive statistical summary of our data. So it's going to print out the mean, max, min, median, and then like various percentiles of the data. So it's going to give us that quick view um of all those statistics, which is really nice. So let's let's see that. So if we print out the stats, we can see what everything that gets produced. So you can see here we get um this summary. This is our descriptive summary. We get a count of how many values are in the series. We know that there's only five because we created it with only five. That makes sense. The average is three. That also makes sense. Just the numbers one through five. So the average is three. The standard deviation is 1.58. The minimum is one. 25th percentile is a two. The median which is the 50th percentile is three. That makes sense. The upper the 75th percentile is four. Also makes sense. And the maximum is five. That makes sense, right? It's just 1 2 3 4 5 is our data. But that's extremely useful, right? To get that quick, you just run one function, you get all this summary of the different the various stats of that data is pretty useful and and it's actually going to be incredibly useful to do when we have data frames. The describe is going to be something we'll use quite a bit on an entire collection of series in the data frame. Okay, so if we want to get all of the unique values of the series, we can run this unique function which will uh give us all the unique values that are exist within the within the series. So if we um print this out, this should be the values um one through five because we don't have any duplicates. They're all unique, right? So this should just give us um all of those values. Clear that out. And it does, right? So 1 2 3 4 5 um this basically gives us a list or or an array essentially of all the unique values if we if we run the unique function on the series. That's another advantage, right? If your if your data is in a series, you can quickly get all the unique values. If it's in a if it's in a dictionary, that might be harder to do. Might be harder to kind of sort through it and and try to um dduplicate it. So unique um we can also get the number of unique. So if we just stick a n right in front of that unique uh n unique gives us a count. So this should be five. This should tell us how many unique values there are. Um we should be able to print this out and see that there are five total uh unique values. Right? The just the numbers 1 2 3 4 5. Fantastic. So we can keep going then. So just like with um numpy arrays, we can do arithmetic with series. So we can add series, we can subtract series. The catch being that it can be tricky to do arithmetic with series when they do not have the same index. Um let's see how that plays out. So these series have different indices. Remember this series was built with the default index with 0 1 2 3 4 5 sorry 0 1 2 3 4 um as the as the indices and this one was built with the indices that uh have the the the character indices right a b cde e so it might be a question of like what actually happens if we add those series together like the data if we were just looking at the data arrays those make sense to add together because they're they're just arrays that are the same shape. So it makes sense to add their elements together and and that should be a very natural thing. The question is what happens with their uh indices. So let's see let's see what we get. Let's print that out. So it turns out that pandis doesn't know what to do with that because they don't have the same uh the same indices. So this is just a warning that you can normally you can add together series that have the same indices no problem and the resulting data is just going to have that same index as the first two. But when you add together these have different indices. So just to give a warning here we have a warning. These series have different indices. um we can't add them together. Okay, we can't really add them. Now, we don't get an error, but we basically get uh we basically get undefined result, right? It we don't know how to add things together. And look at what the new index is. It it thinks it's adding them two together. We get 0 1 2 3 4 and then abcde e. It's just a mess, right? It's just a mess. It doesn't really make sense to do. So if you're adding series together, doing arithmetic in general, multiplying, subtracting, um, all those numpy arithmetic operations we learned last week, you can do them as long as they share the same index. The index order matters because it that dictates the position of elements, right? The relative position of elements is determined by the index. Okay. Now, I want to show you something really cool as well, which is the the ability to basically apply a function to every element of the series. So, if we have a series, we can manipulate every element of it at once by using a special function called apply. So, apply does exactly what it sounds like. It applies a function to every single element. So, the input to apply is going to be a function. So this is where the the input the or I should say the argument to the apply function is a function and we have a special function here that we have not seen before called a lambda function. So we haven't seen this before. What this is this notation here actually is declaring a function. Lambda declares a function uh without having to use defaf. So it's a it's a shorthand in Python to do uh to define a function on one line of code. And what it means is we this is defining a function very similar to defaf and it's saying we have a function that has um one argument and we take that argument and square it. So so that's a function. and it's saying take and take whatever argument is input to this function and square it. So what that means is on this series we're going to take every element because those are the arguments to this apply and we're going to square them. So that's what this means. So this has the effect of squaring each element because it's uh star two remember is a second power. So this will square each element. That's what apply means is apply some function to every single element. And this is a we're defining a function here to say take every argument and square it. That's what that function is. Now do we have to do it that way? No. We the alternative to doing this is if we quickly came up here and said let's define a function called square where we take any x and we return x star 2. That's a function really easily, right? That's a function that squares the thing the arguments. And so we instead of doing this, we could alternatively do the same exact thing, but instead of passing in this lambda, we could pass in our uh square function there. Okay, which says we should be taking the elements of x and uh the elements of the array, sorry, the series, and we should be squaring them. This is the same thing. This is the same as above. It's the same thing as doing this. Lambda is just a shorthand for defining just doing defaf and returning this argument. This is your argument to the defaf and this is what you're doing to that argument. You're squaring it. Does that make sense on the lambda that this this is equivalent to this is exactly equivalent to this lambda. Exactly equivalent to that lambda is just a shorthand for this function. Roberto yeah it's shorthand. So lambda notation is you do you declare lambda as a keyword and then you um write down your argument and you can actually have more than one argument. So you could do x comma y comma z if you have three arguments. Let's say in the series we only have one argument which is every element. That's the intention is to apply this function to every element. So there's only ever going to be one argument. So we have one argument and then we have a colon to and then we have our math to say what are we going to do with the argument to return. So the colon is like shorthand for return. Lambda is like shorthand for defaf. So yeah it is it's always deaf returned. >> Okay. >> We always so the other thing I don't want to the other thing I don't want to get away from either is this notion of apply. This is a really powerful idea. This is a nice shorthand to say I want to take every single element of this series and there could be millions of them. There could be millions of these elements. I want to take every single element and apply some function to every single element and this is the way to do it. So apply is really powerful. Uh we're actually going to use apply quite a bit as we move along and work with data frames. Apply will be really useful to apply a function to every column or specific columns that we want to transform. We will use apply. Okay, let's make sure this works. Let's print out. So let's print out the squared series. Let's do both. Let's do the lambda version first. So let's uncomment that. So see how this uh let me sorry. Let me comment this out so we don't try to print that out again. rerun that. Okay. So, see how it ended up squaring everything. So, there we we took the apply and and gave it the lambda function and and ended up squaring every element all at once. So, everything gets squared. Um, and again like that's we could do it this way. Um, alternatively, we could run it this way and it should work the same. There it goes. So, so, so the same result whether we use the traditional deaf function uh definition or we use a lambda either way the same thing. Okay, cool. So, no questions on apply. We we will use apply quite a bit whenever we want to make sure to transform all elements of a series. Remember in the apply every element of the series is going to be applied to this function or I should say this function is going to be applied to every element of the series. So apply will will transform everything in that series according to this function. Now the kind of the more targeted way to replace values is is the series map. So this is a little less um this is basically more targeted than apply more targeted than apply. It's it's a um replacement of um values uh within the series. So, um, essentially what we're doing here is we're going to take our series, replace this data with this data here, replace this two that's in our series with this data here, and replace this with this data here. And so, you can choose which data values you're replacing in the map. You don't get to choose that with apply. apply is going to apply that transformation to everything in the series. All of it's going to be mapped with this function. Whereas map is going to do a specific replacement. So map can be useful like if you only want to transform certain data values. So if you want to find out where there's a zero or a one or a two and replace those with certain values, you can do this. This kind of does like a find and replace. You think about it like that. Find and replace. So let's see what the result of this is if we print the um mapped series. So you can see that uh one two and three get replaced but we actually uh don't have any replacement for three and four. So those uh get mapped to nan because we don't have a specific replacement for them and their original data was integers. So we now have strings and we had those integers we don't know how to replace those. So those get overwritten with nan. So the thing you have to be careful with a map is you want to make sure you basically cover all the cases that you have um within your series. So if you have, you know, 1 2 3, we know that the other data members are four and a five. We should probably pick some candidates to replace those with. So if we have um four, we want to map that to four probably and we had a five, we want to map that to uh five. So again, this is a more specific find and replace. So now all of that data has been replaced and we actually have um those replacements uh that that we can now make. Okay. So map will map these data members to this value. And that will happen for every copy of this. So if we had a bunch of ones, they all would be replaced with one. Every instance of one would be replaced with one. Not just like the first one, but every instance. All right. questions about map. Do we see how it's different than apply? Apply is going to take a function and apply it to everything all at once. We're not picking and choosing what to replace something by, which is what we're doing with map. All right. Very good. A couple more examples. So, beyond applying transformations or replacing certain values with uh with something, we can also do things like sorting. So, this should make sense to us that we can sort. So, by the way, the default sorting is uh default sorting is least to greatest. So, it's an ascending sort. If we want to turn that off and do a greatest to least, we have to go into here and turn that um we have to do ascending to false. Okay. So if we do ascending to false, let me jot that down. Ascending equals false will do greatest to least. So this will sort our data. So we have 1 2 3 4 5 currently from the beginning to the end of our series. If we do this sort um it's already sorted, but if we do a greatest to least, it should um put it now as 5 43 2 1, right? It should now be sorted like that. So let's go ahead and print that out to be sure that it got um sorted properly. So print sorted series. And you can see now we have 5 4 3 2 1. And you can also see the indices now change to where this is now 4 3 2 1 0 because the data retains its index index. So we're sorting the whole series. The index also gets sorted along the way. So um that's something to be aware of is when you're sorting the series, you're sorting both the data and the label. That label comes along with it or I should say the index, not the label, the index. So if we were to keep this as true, which is the default behavior, uh it would just be the the what how we had it was already sorted, right? 1 2 3 4 5. So in order to do a greatest to least, we have to put this to false ascending as false and that will be the the that will be the greatest to least sort. Does that make sense? Any questions on the sorting? That that's valuable, right? That should be a natural thing we can do is we can sort the series. You can do that in any uh spreadsheet, right? You can sort a column. So we better be able to do that, right? that that should make sense as a a basic operation we should have is the ability to sort a series. >> Okay, >> very good. Let me move on to a couple of really important things that deal with missing data. So, we have a a simple function here to check and see if there's any missing values um which is series null. So this will check and see if there's anything missing, meaning that we have a blank value or an N value in our data. So we're missing something for that index for some reason. Maybe we loaded in this data from a spreadsheet and we're missing a value. We want to be able to see if we actually are missing anything. Let's see what this returns. So you can see what this this does. This goes through and checks every entry and sees if sees us if if any of them are true or false if it's true. So this will return this will return true for missing entries. So if there was a true here that means that slot in the series has a missing value. It has an nan. It was blank. Essentially it has an it has an nan. it doesn't doesn't have a valid value there. It's null. So, uh this will give us a check for every every entry. And um you know more importantly than that is maybe we want to know the the total of them. So let me show you an additional example is we can see the number of missing values which is if we just tack on a sum if we just tack on a sum there at the end. So if we take this and total that up as an aggregation we can we can get how many missing values there are. So this should be zero. There is no missing data in our sample series, right? It's just one, two, three, four, five. There's no missing numbers here. So, but we could uh confirm that this should be zero. So, you see here we get zero as the number of missing values there. There is no so so both of these are valuable. Um this is valuable to see which slots are missing. uh and we could uh have a place to see how many are missing. Can you insert a null? Yeah, you can overwrite. So we could overwrite a value to be null by doing something like uh series um zero equals none. That would overwrite the first entry to be to be a null value none. Does that make sense? This would this would overwrite the first entry to be null. Yeah, none. The the Python equivalent is none. Yeah, by the way, if we do that, look how many are missing now. After we do that, look how many we get missing now. We get one, right? So, we over we overwrote that first item. Now, when we do this, we actually get one. That should make some sense to us, right? Yeah. It's because count is count is usually counting the um number of a of a specific data value like count how many zeros there are, count how many ones there are, count how many twos there are. In this case, we're getting a total. That's what sum represents is how many total nles do we have. So we think we're thinking of count a little differently in this context. like it's they count how many of a certain uh data type we have. This case we're we're checking is null which gives us like a boolean zero or a one and we're totaling that up. So if we have all zeros the total if we add a bunch of zeros we get zero. All those falses are like zeros. So we total all those zeros we get zero. In this case, you know, when we fill in a null value here, we get uh we get one missing value as the total because now there's a there's a true. There's actually a true sitting here. Um which we could see if we we could see if we printed that out again, if we printed uh series.isnull, um there's now a true sitting here, which is a one. Basically, Python treats as one. So, we total that up, we get one. All right. So, you may be asking, well, what do we do if we have a null value? Well, luckily, there is a nice function to replace null values with a default value, and that is the that is the function fill na. So, if we're missing a null, if we're missing something, we can go ahead and just fill it with a specified value. So, in this situation, we have a null here. Let's fill it with a one. Let's go ahead and fill it back in with a one because that's what it used to be before I took it out and and said it was none. Let's go ahead and fill that NA with a n with a one. And then let's print the filled series. So now this should have that one. This should have that null value filled in with the one. And uh there we go. So we have one there and all the values are we have no more null values. So fill na will uh fill back in uh it will fill in any missing values with a default. Okay so pretty convenient. If we identify anything missing we can fill it in right we can fill it in with a default with this fill in a. All right. So, what I wanted to show you next is the ability to essentially filter a series. So, we can put in conditions to to basically filter out uh different rows or different entries in the series. And I want to show you how to do that very easily, which are just going to use our comparison operators to basically um generate some filters for us. So, I want you to see the syntax of that. Um so we have this sample series that we built from this uh fake data. We have uh index a b cde e and we have that data mapped or we have those values mapped to this data 10 20 30 40 50. Okay. So some fake data there and we build our series. So we we have a new series sitting here. I want you to see the syntax of how we can generate a filter. So generally we can build filters by by using the comparison operators and the brackets for value selection. So remember we typically will use brackets to put an index in there. That's typically what we're used to doing, right? we put in a zero or we put in an a or we do a slice, we can grab elements that way. So think of doing that but with a comparison filter. Now so this would like for instance this says we want to pick only elements of the series that are going to be bigger than 30. So we have uh pretend like we were going to pass in an index here. So we have series bracket because what would what would typically go here would be something like this. Right? That would typically select something from the series. It would select the first item or or we could slice it to be something like that. Right? We could slice. So that's how we select things. But in this case, we're actually going to do a filter, which is a condition to say I only want to keep items that meet this criteria. That's the filter. Okay. So inside the brackets is our filter condition. So it's basically um we only keep elements that meet the filtered. So let me show you what this should be. If we're filtering out to say only give me items from the series that are bigger than 30, we should only get this part of the series remaining. Right? Everything else should be filtered out because 10, 20, and 30 are not bigger than 30. They're all less than or equal to. So these guys should be left out and we should only be seeing this as the result of that filter, right? We should only see those guys as the result of the filter. So let's make sure we do. Let's print out this selected greater than 30. So here it is, right? We only Oh, it already did it for us. So it already showed us this the selected greater than 30 results in only those two uh only those two um parts of the series remaining. Um so that that was an effective filter, right? To only give us the items that are bigger than 30. So that's how we filter. And so all we have to do is if we want to do other kinds of filters, we just have have to do other kinds of comparison operators. So for instance, let's pick all the items of our series that are exactly equal to 20. How many of those do we have? We really only have one. Only this guy exactly equals 20, right? So So when we do this filter, this should ignore everything else and only give us this item out of the series. So our series is going to be filtered down quite a bit to only this element and we can see that. So when we come down here the part that um gets printed out there is going to be only that part of the series the the index B and the 20 the data 20 right so that was that is that filter not equal to 40 which items of our series are not equal to 40 that's what this filter really is asking so we go to our data and look at it we have a 40 here which parts of the series are not equal to 40 well it's basically everything else Right? This is not equal to 40. This certainly is. This is and this is as well. It should be everything but that 40 should be filtered out and returned back to us. So we if we take a look at that what is not equal to 40 we get everything else in the series 10 20 30 50. So that was an effective filter exactly what we were hoping for. Okay. So, greater than 30, equal to 20, not equal to 40. Those are all valid filters. And I want you to see that syntax where this is inside of the bracket. It's it's basically replacing an index. They say I want to pick something based on a condition, not based on an index. Okay? Based on a condition. Okay? So, I'm going to show you some other uh other examples of conditions. For for instance um we can do we can actually combine conditions using logical uh shortorthhands. So this is two multiple conditions we're combining it is greater than 20 but less than 50. So this is greater than 20 and this symbol here we haven't seen yet is the shorthand for and it's the boolean and symbol. So this amperand symbol is um shorthand for and. So this means that we're looking to filter out data that is bigger than 20 and less than 50. So multiple conditions combined. Now we could do or. There's a shorthand for ore that we could use in here. And that is the uh pipe. Basically the the pipe uh symbol is shorthand for the logical or so we could have a um pipe symbol here. Oops, I don't know what happened. Yeah, we could have a pipe symbol here which would signal that we would be looking at this condition or this condition. So things that are less than 50 or bigger than 20 which would really be everything. So it's kind of a useless condition. But we can have that one. So does that make sense? That those are just shorthands for and and or. And so we can because we can filter on conditions. There's nothing stopping us from filtering on multiple conditions that are logically combined. So greater than 20 and less than 50. Well, what what should that be? Should really just be these guys here. Those are the only things that are greater than 20 and less than 50. We should just get those two out of it. So if we went down and looked at it, we would get these two guys as the filter, which makes sense. So we can do that. That's a valid way of filtering. You can also, what's really cool is you can also filter based on a specific list. So you could say I only want to keep values that are inside of a specific list. So this is where we use the is um function. So this is a function that gives a boolean uh true or false if if the members of this data are inside of this exact list. So what this means is we should really only be keeping values that are going to be 20, 40 or 60. Only those three numbers should be kept. So if we go back to our data, we only are going to keep these ent only these two. There's no 60. So nothing 60 wouldn't match to anything. So we're really only going to keep the 20 and the 40. So is in is a special kind of filter that um only retains values that are actually in this in this collection. So if we did this by list, we only get those two only the 20 and the 40 remain. You can do string uh filters. So for instance, if we had our data was all strings, we could um filter based on the string uh starting with a B. So that's a that's actually a very specific condition, right? So we want to filter all strings that only begin with B. So in this case, that would just be the banana should be the only one that that is returned, the only entry that's returned. Everything else starts with um A, C, D, or E. So that's possible. Okay. So what's really important to know is that LO uh allows us to pass in the basically the labeled index. So our our index that we have which is either the default one or the one we provided when we created it. In this case we created a series that had a index with a b c d and e. So lo will allow us to select multiple of those indices almost like a slice. So so lo allows us to use our index. So we use our series index to select data. So we're selecting A, we're selecting C, and we're selecting E. So we pass those into a list inside of the LO function. And um we we can pass in uh as many as we want. We can pass in uh just a single value. It doesn't need to be inside of a list. But but LO is pretty typical for selecting multiple elements uh using the index. So that's the key thing to remember about LO. It uses the index of the series to select items and you pass in a a typically a list of those indices that you want to pick. So we're picking the elements at A, element at C and the element at E. Let's go back to our data. That should be 10, 30, and 50. Um so if we go back and look that should be at uh those three guys right a c and e 10 30 and 50. Okay so lo uses our uh indices and now how is that different than iOS so iO uses position. So this always uses position regardless of index. So even if you have a really nice index that you've provided like a date or in this case abcde e we can ignore those and just use their position within the series which is what iO does. It just uses the numerical position of the elements. So um for instance we want to pick the very first item um the very first item we know is at index a but in terms of its um position is really just position zero within the series right position zero is the very first item even though it has index a in this case we're picking the positions one all the way up to four we're doing a slice we can do that we can slice based on position so that's okay with iOS is different than LO. LO uses the index whatever index we have defined which is um it can be special right it can be dates it can be strings it can be objects um that index can be whatever we define it to be will always use numerical positions always so you would never put in like this would be invalid to do this this would give us an error we can only use this with lo because that is the index of the uh series, but the positions are just the data positions within the series array. So we have um just a natural ordering that always exists of our data within the series and we can use that with okay so very um they're very similar. They're both used for selecting data but they use different indices. LO uses the the defined index of the series and I look uses the numerical position of the elements. Okay. So we see by the way this is one all the way up to four. So what is at position one would be this guy even though it's at index B. Even though it's at index B. And what is position four? Well this is 0 1 2 3 fourth. The fourth element of our data. Um so that would be all the way up to 50 but not actually including 50. That's the slice. So it should be uh 10 20 30. What is four? It's a position. It's just a it's it's this would represent the fourth element in the data. The fourth element in the data which is uh this is the zeroith element. This is the first element. This is the second third. Third and this is the fourth. Right? Numerical position. That's what I look uses. Now what data do if you so if you were to ignore if you were to ignore this index for a minute this is the index and just think about this data's positioning from left to right what index would we say or what position would we say is the 10 at what position is the 10 in the array if we were to just think of it as a list let's say what position is that right it's position zero and what position is 20? It's at position one, right? And so 30 should be at what should be what should be 30 two perfect 40 should be three and 50 should be four. So what I'm what I was saying with those examples is I look uses these positions while LO uses the index. Does that make sense? Now you think of it this way. I look uses those positions of the data, the relative positions of them to select things. So when we say one to four, that's this slice, right? That's this slice of the data. Uh not including I should say not including the four. So it's really this slice. Okay. So what I wanted to show you guys is that was the end of this notebook. But there is some extra practice. Um I encourage you guys to try this out on your own. So this is your own uh practice here at the end. Try this out on your own. So this 4.01 notebook. Try this out when you get a chance to do some extra practice accessing and manipulating this data. Okay. So, you have some example uh arrays here. You'll build some series and uh go through and do some uh selecting of some data using uh using the uh different methodologies we just showed with lo and uh picking different indices and things like that. So this will be some extra practice for you. What I want to do now is go over to talk about the data frame. So we talked a we've talked about series so far. We have some different functions on a series. These most of these are going to carry over exactly into the data frame in the two-dimensional aspect. So everything we've been doing mostly been dealing with onedimensional data, but we are going to deal with two dimensional data. Let's go over to 4.02 now and talk about the data frame. Okay. So if you have a moment, pull up the 4.02 notebook. We're going to go over to that and talk about the data frame now which is going to extend everything we just learned over into the into the two dimension two dimensional data with rows and columns now which is going to be the data frame. You guys have this notebook and go over to the 4.02. So let's remind ourselves what a data frame is. So it a dataf frame is now going to be our two-dimensional extension of the series where we are going to have a tabular data structure. Meaning it's going to look more like a table. We're going to have rows and columns to deal with now. And what's going to be really interesting is we'll actually have multiple indices to worry about because we have multiple dimensions. So imagine if we had these two series of apples and oranges. So we had a series called apples. It has this data 3201 and it has just a natural positional index of 0123. So nothing that interesting. We have another series over here called oranges that has data that looks like 0372. It also has the same the same uh index of 0123. That's fine. That's just a default index. Now, what we can do is concatenate these into a single data structure called a data frame where now these two series actually form columns in this table structure. So now we have more of a table which is a data frame that contains one row index. So this is called the row index here. So it has a a row index which is row zero, row one, row two, row three. So we can think about the data at a row level. So we can say okay this is one row, this is one row. It's very much like a spreadsheet, right? This is one row. This is one row. We have that. And then we have um then we have uh data at the column level. So we have these what are known as the column index. Column index that we have here. So the column index just allows us to select columns based on those names. So we can always go into this table and grab the apples column. We can grab the oranges column. We could grab the apples and oranges columns together and pull back multiple columns. We have that flexibility with a data frame just like you would in a spreadsheet. You can pick data at a column level as well. So you always have that uh always have that flexibility in a data frame to select rows to select columns to select rows and columns. So that's something we're going to learn how to do is select data based on where where it is, what row it's in, what column it's in. Um select sections of a data frame or rows and columns will be the merge title. So that's up to us as a variable. The the title of the data frame is not that important. It's more um when we add these two columns together, we get a single data frame with with two columns and we have a row index and a column index. Oops, this is the row index. But the name is whatever we whatever we want to call it. We can label it whatever we want as a variable. Like we can say this is our this is our fruits data frame. Just call it that roots data frame which has apples and oranges as columns and it has four rows right it has four rows of data and it has two columns. Basically the shape of this is a four row two column matrix right think about this as a numpy array. That's what that is right four rows and two columns. Okay so one thing to point out about this that so I I told you guys it has a row index and a column index. So those will be really useful for when we want to select data from the dataf frame. But the other thing is there's no restriction on the data types in the dataf frame. So the data types of one column can be completely different than the data types of the next column. So like this column could be all strings, this column could be all floats. Um that's totally fine. There's there's no uh there's no restriction there of what the columns can be. They can be uh mixed data types. Um that's totally fine. There's no restrictions on the data types. Oh, that's just random data. That's just random. It doesn't represent anything meaningful. It's just random. We just wanted some values to be in there. Yeah, they can be different for each row. Yeah, they can certainly be like they they uh generally generally could be if let's say I had uh like this data type was a string um so this column was a string this column was a float then I'm going to have like within this row this is going to be a this is going to be a float but this is going to be a string so yeah the rows can be different if you have mixed data types the rows are going to be different so the most meaningful example I can give you is most spreadsheets can be loaded into a dataf frame structure. So if you take any like CSV or Excel sheet, they can be loaded into a data frame and then we can do all of our analysis on that data frame. That's typically what we're going to do with the data that that is inside of a a commaepparated value file, a CSV file. We'll load that into a dataf frame. We're going to practice doing that quite a bit. Okay. All right. So before we get to those uh examples, let's talk about how we can create some example data frames um using our code. So here we are going to import pandas as pd. And one of the most common ways um creating a data frame is from a dictionary. That's the most one of the most common ways of creating a dataf frame is is from a dictionary. Now why from a dictionary? It's because these keys in the dictionary are actually going to be our column names. These are going to be our columns. The the keys are going to be our columns. And then the the data within those columns are going to be inside of this list. Okay? Inside of each of these lists. So I'll show us that data frame in a moment. But um this is pretty common. So the uh keys become the column uh in indices and the data within the columns are the values which are the lists. So those lists become the data within the columns. So the name has these three strings, the age has these three ages and the salaries have these three uh integers. So let's go ahead and see that dictionary. So it actually looks like this. Name, age, and salary. It looks like this. This is our data. Um, it has three rows. That's not surprising because it has three every column has three entries. So it ends up being three rows. And here's the name column has Alice, Bob, and Charlie. The age column has those ages that were in that list. and the salary column has those those three there. So using a dictionary is one way of building a data frame. It's a very common way. Um so we can do that. Um the other way is to use mixtype list. So basically where you define every row. So creating a data frame from list is possible where we have a list for each row. So we have the first row, we have the second row, we have the third row and we actually manually supply what the column names should be. So this is all the rowle data and this is the column name. So we can actually build a data frame off of that using pd dataf frame. We pass in our row data and we say here's what our column should be labeled. Here's the column indices. So these are the column indices. We say here's what our column should be. Here's our row data. And that builds the same data frame. Builds the same exact data frame as we had before. Just doing it not in a dictionary but in a in a list uh list of lists essentially, right? All of our rows are utilized here as as like this. But this does build the same exact same exact data frame. All right. Same same thing we can use numpy arrays. Now this is a numpy array of this is the 2D array of our rows right so here's the first row here's the second row here's the third row and we can give it our column indices as well so again the numpy array data numpy array data is our rows of the data frame. So uh this data formulates our rows and we're passing that in to build our data frame. Uh it so this should be the same exact data frame that we have from before just using numpy arrays to build it. Okay. All right. Let me give you a more practical example that will be realistic to a lot of the use cases we will see over the course of the program when we're loading in data. And this is one that's used all over the industry. Um people use it all the time. I I use this one nearly every day is reading data from a file. So if you want to read a CSV file, you use the PD.ED CSV function. So read CSV and all you have to do is pass in the file path. So um now this path looks a little wonky because I am inside of Collab and I had to mount this file within my drive. So this data is in my Google drive on Collab. But generally that like if you're in VS Code, this would be a path to um the data that is uh somewhere on your computer. So this is just a file path to this CSV file. Okay? And by the way, you guys should have this data. Um if you go into your Let me let me show you where it is. So if you go into your LMS, let me log in. Do you do you guys have access to the data sets? It should be from your reference materials. It should be in this data sets here. Okay, this data sets. So go to this data sets and the reference materials and that's where I'm getting this house price uh CSV from. So if you download that and extract it on your machine um that's where you can get that that's where you can get that file from. So, house prices. Um, by the way, if you have it in your Google Drive, you have to run this code. Um, you have to run this code first to be able to access uh files that are stored on your drive in in Google. So, so you want to run that first and that makes your data accessible within your within your So, this is if you're running in Collab. If you're running in collab, you want to run that uh in a cell. So you can This makes your data accessible. Let me write that down. This makes your Google Drive data in Collab. Okay. So you have to run that and then once once you run that then you can access data that's within your drive. So, if you go to this like folder icon, you can see you can you can you can browse your drive uh within this folder icon. This is likely going to fail for me because I didn't mount the drive yet. Yep. So, it can't find my drive files, but I could I just run this permit. Uh, there is, but only for the folder you have open. It's just the like file explorer on the left, Roberto. It's just the file explorer on the left, but only if you have uh it only in the folder you have open. So, what I would recommend if you're working in VS Code is to I would recommend to put that data set in the same folder as where your notebook is so it can discover that file pretty easily. Does that make sense? Like put that put that data set file. So, there's two files in this uh notebook that you're going to want access to, which is the house prices and the Iris Excel. So put those in the same location as this notebook and it should work. So let me actually put a comment here. So uh you should be doing PD read CSV and then it should be file path to CSV wherever that is. So, I think the easiest I think the easiest thing to do is put it at the same location as where the notebook is saved because if it's in the same place, um, then that should just be PD read CSV and then it should be um, house price.csv should just be that loads the whole file. Yeah. Yeah. Loads all the data in memory. I don't have import drive command on the Yeah, this is for Google. This is for Collab. This is this is only for Collab. You need this to access data on your on your uh Google Drive. They the two files are the house prices CSV and the Iris Excel XLSX. Do you have those two? house prices CSV and Iris XL XLSX Excel sheet. So you can build a data frame from here. Now when you uh load all this data in, you print it out. It's actually quite a big file. It has about 5,000 rows. So it's got it's not going to show you all of that data, but we're going to have some functions we can do that can easily get us some summaries of of this data. um which will be nice. Okay, so we'll have that coming up shortly. But there are 5,000 rows. So or sorry about 4,600 rows. So it it's a you know decent amount of data there. 4,600 rows. And the iris data also has 460 rows and 18 columns or 4,600 rows, I should say. Sorry. All right. Were you guys able to run this? Were you able to uh read the CSV using pandas? Were you able to run this? Not able to. Uh make sure that the path you have the Oh, that's fine. The output's getting truncated. That's fine. That's expected. It's not going to show you all 4600 rows. That's that's expected. Yes. Are you running in Collab, Mariel? You are. Okay. Uh you have to give permissions. So there should be like a popup when you run this code. When you run this, there should be a popup to give permissions from your Google accounts for this notebook to access your your files. So make sure that that popup actually shows up and you click and you uh validate it. You should see a popup when you run this. Okay. Uh worst case, by the way, um what you could do if if your drive isn't mounting a backup is to go do you see this file? If you see this files on the left, um, and this is for for everyone running Collab, if you're unable to get the drive to mount, what you could do is you could just upload the file into the workspace. So, if you go to this files, you can upload, you can hit this upload button to upload data. So you can see see this upload here from if you click on this and then you click on this you can upload the file manually into the into this uh workspace into the notebook workspace. So you could you could do that too if you have the file in your computer just upload it. That works too without having to mount your whole drive. So just click on this upload and then you can upload the you can upload the file. So, so I just uploaded the house price CSV. So now, now I have it available in the session and then I could like that. What should it look like? Uh, it should look like this. You don't have to you don't have to do Google Drive. You can do it this way too. Do you see the way I just showed where you can uh upload it into the session into the notebook session? You can do that. You can keep the file on your local and then just upload it to your notebook using this uh files upload. Roberto, do you see this like all this data displaying? That's how you know it worked. Uh if you rerun the cell, it that means it worked. It should it should display the same thing. Uh if you scroll back up, uh people posted it, but it's in your LMS. It's in the in the data sets here. See this data sets link? It's this one. Download the data sets. And uh it's we're doing the house price and the Excel and the Iris XLX. Okay. So, were people able to run this and get the you should see this output that has a bunch of the So, so this is printing out the contents of the CSV. That's right, Roberto. Yep. Yep. That's right. Okay. So you guys should be able to see this. Now this is um loading. Let me just take a step back and say this is loading the data into a data frame. So now this is the the contents of the CSV are now in a dataf frame. So we can do dataf frame operations to that data which is going to be really useful. We can manipulate it. We can do summaries. We can do group buys. We can do all we can do filtering. We can do all kinds of stuff to this data frame. Now that we've loaded all that data into the data frame. So read CSV we are going to use a lot during the program. We're going to use a lot to load in data from a file to load in data from like a spreadsheet style file into a data frame and then manipulate that data frame. Okay. And there's there's a read Excel. Pandas actually supports a lot of different read functions. And so it has read JSON, read parquet, read all read um arrow, it has all kinds of different file types that it's uh that it supports. Okay, so uh not just CSV and Excel, even though those are very common, you know, we can read all kinds of different files in pandas to load them into a data frame as long as they're supported. Okay. All right. So, we uh practiced loading in that data, but now what we want to do is practice accessing data from that data frame. Now that we've seen we can load data in, how do we actually access data and then start to manipulate it, summarize it, do those all all those things we were doing with series, we want to extend into the data frame. So, let's get some practice there. Um let's scroll down and build an example data frame with some uh fake columns here. So we have column name column one column 2 another column. We're going to practice accessing data from this and then eventually we're going to come back to this data frame with all that uh data we just loaded from the house prices or the iris and practice working with that. But for the moment let's work with this fake data frame built from this. So we do pd.dataf frame that builds that builds it out of this dictionary. And here's our here's our column data that we have uh mapped to these different column names. So if we want to access a column, it's super easy to do. We just pass in that column name here inside of the brackets. So we do df, which is the name of our data frame, and we do um we just have column name here. that's going to access all of the data within a column. By the way, what do you guys think is the data type of an entire column? We just studied it. What do you think an entire column is inside of a data frame? It's a particular type of object. You're right. It's an object, but what what type of object? It's a pandas object, not a string. We just learned about it. We just were manipulating it. So, a column is a series. Okay. A column is an entire series. Yep. A column is a series. So when we access this column, it's as if we're accessing an entire series. Okay. So if we print out that column, we get this. This looks like a series, doesn't it? It looks the same because that's what it is. It's a series. We're grabbing that entire column, which is 5, 15, and 8. So to access a single column we just use the braces and we pass in that column index name and we pull back all the data from that column which is really a series. So remember um a single column is a series is a pandas series. So we're actually accessing that series when we do column name. By the way, if we were to try and and access an invalid name, uh we will get an error. So let's say we did um some column. Now this doesn't exist. So this this will give me an error. So if I try to do this and that doesn't exist, this will give me an error. And you can see what kind of error I get. A key error um that this does not exist. Basically, I don't know how to access that column because it doesn't exist, right? So, this is uh we would not want to do this. We would want to use the proper column uh name and that should work just fine. Okay. All right. Now, how about this guy? When we pull back multiple columns, so we can use a list and select multiple columns at once. Now what is a collection of series? We are studying it right now. What is a pandas collection of series? So when we pull back multiple columns, this should be multiple series which is a data frame. So you guys are absolutely right. When we access multiple columns gives us a data frame. So, this is actually going to be kind of a miniature data frame. It's only going to be these two columns because we're accessing these two guys, which are these two columns worth of data. That's kind of like a mini data frame that's part of the whole data frame, right? Selected columns there. So, let's look at that. When we access multiple columns, we get these two guys, which is a which is a data frame. Okay? So we get column one, column two and we see the data and we have three different rows from from the data frame. So we definitely can access as many columns as we want by passing in a list into that uh into that dataf frame uh bracket and it will allow us to access uh multiple columns. Now we can't forget about LO and eyel. So this is saying let's grab the data that is at row zero. So again, this is using the positional index zero. Regardless of what the actual index is, um the userdefined index and data frames can have userdefined index. The rows can actually be a custom index as well because we know a series can have a custom index. So doesn't really care about our custom index. It's going to use the positional index to grab that entire row. It's grabbing the first row. So, df.lo is grabbing this entire row which has values of five, 10, and 100 um and 25. If you look at it, that makes sense. That should be this row here. That's the row, that first row. We can do filters just like we did with series. We did filters. We can do um so we can do conditional uh filters here where um now how do we do that? It's exactly how we would have done with the series right so we do a bracket to signal that I want to access something and here we give a filter. So what this filter does is this will filter rows of the data frame that meet the condition. Okay. So any row where this condition is occurring in other words this column name it has a value greater than 10 we will keep. So we go to that column name bigger than 10. Which ones are bigger than 10? It looks like only this guy is bigger than 10. So we really should only be keeping um we really should only be keeping this row here. should be the only row that gets uh kept because um that is the one that corresponds to where this column has a value greater than 10. Okay, do we see how that filter works? We're really checking. So we go back to the syntax. Go back to the syntax. We're checking where a particular column is greater than 10. only those rows only those rows do we keep where this condition is met. So what we should do is go and double check this. So the filtered rows we only are keeping that one row. So we just filter that one row because that's that's the only place where this guy is bigger than 10. So that's that's very powerful. We do this very big kind of filter operation on the data frame. So you have to think of the filter operation we did on the series. We're extending it into two dimensions. We only keep the rows where this condition is met. Okay? Only keep the rows where that condition is is valid. Okay? It's because overall overall we access data. We access data by using DF and then the bracket, right? So, so that should be generally accessing data. We saw that with the series too, right? It's a series then we have bracket. We saw that with lists. Usually it's a list and then a bracket and we pass in an index. So, so we generally have the first DF with the with the bracket to signal we're accessing something. Okay, it's just generally we're accessing something. Now, the reason we have DF twice is because inside of here is a is a filter condition and the condition says I want to look at this column. So, so we are so the reason we have the data frame tries is you see that we're accessing that column which is df bracket the column this accesses this column and says I want to check where this column is bigger than 10 all the rows where this column is bigger than 10 that's why we see df twice we see it once on the outside to signal that I want to access something from this data frame name. What do I want to access? Only the rows where this condition is being met. Only the rows where that condition is true. All right. I have one more I have one more thing I want to talk to you about on the filtering and then we'll take another um little bit of a longer break. We've been going for a couple hours now. So once we filter like this, we are we're basically picking all the rows that we want out of the original data frame, right? Because that's what this does. it selects rows based on meeting this condition. Um, but there's nothing stopping us from accessing a particular column after we do that filter. So that's what this if I I'm going to scroll down to the bottom of this cell and go to this example and I'll come back and talk about at IAD and LO in a minute. But do you guys see how we are accessing this is the filter again? Same exact filter. This is bringing back. So let me copy this and make a comment here that this part of it um gives us the D dF rows that meet the condition and then then we access a particular column of these rows. So that's what this does. Do we agree on that? that. So this is saying I've already filtered my data frame to be filtered to this set of rows, but it's still all columns. So if I were to draw this, right, if I were to draw this, we've basically filtered it out to this selection of rows. Just imagine that, right? We've filtered it down to this selection of rows. But what this thing does is says let's zoom in on this column right let's zoom in on that column that's what that does so we are allowed to do that in the syntax once we filter we can select a column from the filtered result we can get basically chain those together which is what which is what that's doing that code is doing okay all right so let me yeah let me just summarize the things that we've done uh in this cell. Uh let me move that. Um so all we did was we created a data frame with some example data. We didn't even use that data yet. We'll come back to it, but we're just using this example data from this dictionary. And we were just practicing some different things like accessing a single column. All you have to do is put the brackets and then the column string which corresponds to whatever column you're selecting. That is going to retrieve the series of data. in this case like the the column right 5 15 8. So it's it's doing that. Um you can select multiple columns if you pass those names inside of a list. So column one and column 2. This is actually going to give us a dataf frame result, right? Because now we have multiple series being pulled back. Multiple series equals a data frame. Okay, multiple series equals a data frame. So we have a dataf frame result that is giving us only these two columns out of the four columns that exist. We can use iLO to to grab a row based on the position the positional index of that row like this is the first row. We can do the second row. We could do the third row, the fourth row etc. So that's just the first row. We did a filter. So this is just a basic filter. We were just talking about this where this is the condition and so this is saying let's grab all of the rows out of this data frame where this condition is being met. Okay, all of the rows where that condition is being met. All right, then yeah different from a slice different this is a filter different than a slice. This is spec because this is conditional, right? A slice is more positionalbased. It's not conditional. This this only uh selects data that meets this condition. This is conditional. And then what we said is based on this condition, we can actually select another column right after that which kind of change it together. Yes, it's like a birectional. Yep. Okay. While you try that, uh let's continue talking about um on this uh set of examples here, we had the uh at um we had just gotten to the at. Now the at is like the equivalent of LO for accessing a single entry in the data frame. So this is the equivalent of um DF.lo uh for a single value. So at will use the uh index. So at uses the index uh uses the row index and the column index uh for accessing. So the row index in this data frame is uh just the default. It's just the positional just happens to be that way. Does it have to? No. It could be dates. It can be strings. It could be anything. In this case, we didn't um specify any special row index. Um we we do have column index indices, right? Which are these names. So we can use those, but there's no special row index with this uh dictionary uh when we create this. Um so uh it just has the default row index. So when we use at um we can say I want the value at the first row, which is row zero. the second row which is row one um row the third row fourth row fifth row and those are all going to be this positional index because that is the true index of this data frame the row index however we also have the column index so we can access things based on that so column name we're grabbing the uh element in the column name column but the first row here so this should fetch us this five because we want to be within in this column but the first row. So this should this at um should return to us the uh a five. So if you go down to uh single cell by label this should give us five just which is what it does. So so sometimes you can use at again this you only ever use it if you want to access a single item which is kind of rare. You don't really you're not usually going to be grabbing just a single item in the data frame. usually going to be grabbing multiple rows, multiple columns, but um there is the at function for passing in a row index and a column index to access a single item. Yes, but that's true, Romero, but uh it's I would call it the row index and column index in this data frame. The row index is zero because it has the default index, right? We we it has the default index for the data frame which is 0 1 2 3 it's it's uh this is the row index right this is the row index and then this is the column index column index. Okay. All right. So with at we have the equivalent of LO um but for accessing a single a single item. We also have I at which is the this is the equivalent of eyel. So this is the equivalent of for accessing a single element. And so this one we uh use so this one uses the positional uh index for row and column. So we do not use the name for the column. We only use its positional index as far as which column are we in going from left to right. Are we in the first column? Are we in the second column? Are we in the third? This would signal that we are in the second column because that is the column that is at index one. So if you use the positions with IAT or ILO for example, we would be like this is column position zero. This is column position one, column position 2, three. Regardless of the actual column index, right? It has a position within the data frame going left to right. Every column has a position. So I at uses this column position. It also uses the row position regardless of what this is. It just happens to be the same because the data frame uses the default row index. But so 01 if we did IAT and then we did 0 comma one would be the first row second column right? first row, second column. So that should be uh that should be this guy. So we look into our data that we have which is this data frame right here. Um it should be first row second column it should be 15. So we want to make sure that we get 15. So if we run that single cell by position, don't know why it's saying did we do where did we do? Sorry, row zero column one. So row zero uh column Oh, sorry. That would be this. Yep. It would be 10. Oh, so it just took the value. It actually evaluated the it ignored the column string and and just it evaluated it. Okay, that's what I figured. It wouldn't it wouldn't honor the formula. It would just take the value as a result. Makes sense. Thanks for trying that. That makes sense to me. Is it possible to assign a label to a column and and what? Call it a label. Yeah, you can call it a label. I think um people Yeah. So, so going back up to this, people sometimes call these the label instead of the index like like this column name. They usually Yeah, you can use the word column label or you can use the word column index. Uh either one works. I I think I think people would know what you mean by column label. If that's what you're asking, yeah, you can do that. So, yeah, you can use either of these. They're pretty much equivalent. Okay. All right. Finally, we have uh DF LO. So, if you guys remember, LO uses the uh we we've kind of already talked about this, but LO uses the uh uses the indices, right? So, it uses the row index and the column index. In this case, we're selecting the first row but all of this column named column name. So here we are selecting the first row but all of column name uh column. We're picking all of those values. So we should be getting first row but but the whatever is in the column name entry. And so that would be uh if we go back to we should be in the first row but only getting this five essentially which is the the value in that column name. So lo will use uh lo will use um the indices just like at would. So at uses the indices um lo also uses indices basically lo we would use though when we want to select. So we use log use log just like we did with series right we we use log to select multiple values you can even slice the rows. So for instance we can grab like we can grab the first two rows um and slice it that way and that would pull back multiple values for this um data selected with lo. So it's actually going to grab all of those uh all of those rows zero and one um and the data belonging to column uh sorry column name. So you can even put it so so you can use LO to grab multiple um multiple entries. So you can have a slice in there which is pretty interesting. So here's a slice zero and one rows. Okay. All right. Let me get to some exciting uh summary functions that we will cover next. So I want to go through some basic data frame functions. Now that we've seen how to access data, let's look at some other convenient functions. And some of them are going to be really familiar from basically from series. They're going to be the exact same like head and tail and sort and is null. Those are all going to be the same that we have except now it's going to be in multiple dimensions. So let's take a look at some of those examples. But some of them are going to be pretty unique and but some of them are going to be basic uh multi-dimensional extensions of the uh functions we've already seen for series. So let me walk you through some of the uh functions that we are going to do uh some examples with. The first couple we've already seen which are head and tail. So just like in series we have a head and tail function which which shows us the first few rows of the data frame. It if remember if you don't put anything in the head and tail uh argument like if you leave this blank it will just do by default the first five or the last five rows for head and tail. Now why is that helpful? It's actually really helpful. Often when we load in a data frame, the first thing we typically do is run a head or a tail so we can sanity check what we just loaded without having to print out the entire data frame. Printing out the entire data frame is usually uh inefficient and we don't like it's going to get truncated anyway. So, a very standard thing to do is to just print out the head or just print out the tail so we can um get that view of the first five or last five rows to sanity check what we just loaded in. So, that's a very common thing to do. We're going to see we're going to see examples of that. Um and we're going to use head and tail quite a bit as we move along. And you know, as we load in data, that'll be usually the first thing we do is head or tail just to take a look at the data we just loaded in. Yes, it's going to include the header. Let me show you an example. So, if we go back to if we go back to the data we loaded earlier. Um, so the house prices. Um, let's print out the head. So, df.head. Um, it shows you the it shows you the header, but it just shows you the first five rows. See how it's just the first five rows, but we can see all of the contents of the data pretty nicely. and succinctly. So, so it's a very useful thing to do, right? That when you load in this data, this is very useful to do to sanity check what did we just load? What does it look like? df.head. We can get a sanity check of what the data looks like after we, you know, load it in. And we could do tail. So, um, that will give us the last five in the data set, right? That'll give us the last five. So, we can see, uh, there's 4600 rows. This gives us the last five of them. It's sanity check that, too. Usually, we look at the first first five. Okay. So, head and tail. Now, um, you were asking Roberto about a summary earlier. Here is a really a couple of really nice summary functions, which are going to be, uh, two that we'll use quite a bit. One is called the info, which gives us an information summary of the data frame, including all of the columns that exist, all of the data types, and if there's any data missing from those. So, the df.info info function which you see used right here is an incredibly useful function to give us a summary of what is there. Okay, it gives us a summary of what exists. So, let me show you what that looks like on the uh example I just did. So, on that house price data set, um we looked at d