Beginner Friendly Data Science Course With Python 2026 | Data Science Training | Simplilearn

Key Takeaways

Hey everyone, welcome to our data science with Python full course. Ever wondered how data can transform into powerful decisions? Picture turning a mountain of raw numbers into insightful that drive the future. This data science with Python is your game. Just starting out and looking to level up your skills, we have got you covered. We will kick off things by building a solid foundation in statistics, then dive into the world of Python, your go-to tool for handling data. And along the way, you'll get hands-on experience with large language models, the cuttingedge tech that revolutionizing data analysis. And by the end of this course, you'll not only grasp the theory of statistics, but also know how to apply to solve real world challenges. Plus, we will prepare you for those tricky data science interview questions so you can walk into your next interview with confidence and impress your future employers. So, let's get started. >> Deep learning. Deep learning was first introduced in the 1940s. Deep learning did not develop suddenly. It developed slowly and steadily over seven decades. Many thesis and discoveries were made on deep learning from the 1940s to 2000. Thanks to companies like Facebook and Google, the term deep learning has gained popularity and may give the perception that it is a relatively new concept. Deep learning can be considered as a type of machine learning and artificial intelligence or AI that imitates how humans gain certain types of knowledge. Deep learning includes statistics and predictive modeling. Deep learning makes processes quicker and simpler which is advantageous to data scientists to gather, analyze and interpret massive amounts of data. Having the fundamentals discussed, let's move into the different types of deep learning. Neural networks are the main component of deep learning. But neural networks comprise three main types which contain artificial neural networks or ANN, convolution neural networks or CNN and recurrent neural networks or RNN. Artificial neural networks are inspired biologically by the animal brain. Convolutional neural networks surpass other neural networks when given inputs such as images, voice or audio. It analyzes images by processing data. Recurrent neural networks uses sequential data or series of data. Convolutional neural networks and recurrent neural networks are used in natural language processes, speech recognition, image recognition, and many more. Machine learning. The evolution of ML started with the mathematical modeling of neural networks that served as the basis for the invention of machine learning. In 1943, neuroscientist Warren McCullik and logician Walter Pittz attempted to quantitatively map out how humans make decisions and carry out thinking processes. Therefore, the term machine learning is not new. Machine learning is a branch of artificial intelligence and computer science that uses data and algorithms to imitate how humans learn, gradually increasing the systems accuracy. There are three types of machine learning which include supervised learning. What is supervised learning? Well, here machines are trained using labeled data. Machines predict output based on this data. Now coming to unsupervised learning. Models are not supervised using a training data set. It is comparable to the learning process that occurs in the human brain while learning something new. And the third type of machine learning is reinforcement learning. Here the agent learns from feedback. It learns to behave in a given environment based on actions and the result of the action. This feature can be observed in robotics. Now coming to the evolution of AI. The potential of artificial intelligence wasn't explored until the 1950s. Although the idea has been known for centuries, the term artificial intelligence has been around for a decade. Still, it wasn't until British polymath Alan Turing posed the question of why machines couldn't use knowledge like humans do to solve problems and make decisions. We can define artificial intelligence as a technique of turning a computer-based robot to work and act like humans. Now, let's have a glance at the types of artificial intelligence. Weak AI performs only specific tasks like Apple's Siri, Google Assistant, and Amazon's Alexa. You might have used all of these technologies, but the types I am mentioning after this are under experiment. General AI can also be addressed as artificial general intelligence. It is equivalent to human intelligence. Hence, an AGI system is capable of carrying out any task that a human can. Strong AI aspires to build machines that are indistinguishable from the human mind. Both general and strong AI are hypothetical right now. Rigorous research is going on on this matter. There are many branches of artificial intelligence which include machine learning, deep learning, natural language processing, robotics, expert systems, fuzzy logic. Therefore, the correct answer for which is not a branch of artificial intelligence is option A, data analysis. Now that we have covered deep learning, machine learning and artificial intelligence, the final topic is data science. Concepts like deep learning, machine learning and artificial intelligence can be considered a subset of data science. Let us cover the evolution of data science. The phrase data science was coined in the early 1960s to characterize a new profession that would enable the comprehension and analysis of the massive volumes of data being gathered at the time. Since its beginnings, data science has expanded to incorporate ideas and methods from other fields, including artificial intelligence, machine learning, deep learning, and so forth. Data science can be defined as the domain of study that handles vast volumes of data using modern tools and techniques to find unseen patterns, derive meaningful information, and make business decisions. Therefore, data science comprises machine learning, artificial intelligence and deep learning. >> There are a lot of areas where data science can be used. One of the very common one is fraud detection or fraud prevention. There are a lot of fraudulent activities or transactions primarily on the internet. It's very easy to commit fraud and therefore we can use data science to either prevent or detect fraud. There are certain algorithms, machine learning algorithms that can be used like for example some outlier techniques, clustering techniques that can be used to detect fraud and prevent fraud as well. So who is a data scientist rather? It is actually a very generic role that defines somebody who is working with data is known as a data scientist. But there can be very specific activities and the roles can be actually much more specific. What exactly a person does within the area of data science can be much more specific. But broadly anybody working in the area of data science is known as a data scientist. So what does a data scientist do? These are some of the activities. Data acquisition, data preparation, data mining, data modeling and then model maintenance. We will talk about each of these in a great detail but at a very high level the first step obviously is to get the raw data which is known as data acquisition. It can be all kinds of format and it could be multiple sources but obviously that raw data cannot be used as it is for performing data mining activities or data modeling activities. So the data has to be planned and prepared for using in the data models or in the data mining activity. So that is the data preparation. Then we actually do the data mining which can also include some exploratory activities. And then if we have to do stuff like machine learning then you need to build a machine learning model and test the model get insights out of it and then if um the model is fine you deploy it and then you need to maintain the model because over a period of time it is possible that you need to tweak the model because of change in the process or change in the data and so on. So that all comes under the model maintenance. So let's take deeper look at each of these activities. Let's start with data acquisition. So in the stage of data acquisition basically the data scientist will collect raw data from all possible sources. So this could be typically an RDBMS which is a relational database or it can also be a nonRDBMS or could be flat files or unstructured data and so on. So we need to bring all that data from different sources if required. We need to do some kind of homogeneous formatting so that it all fits into in looks at least format from a format perspective it looks homogeneous. So that may be requiring some kind of transformation. Very often this is loaded into what is known as data warehouse. So this can also be sometimes referred to as ETL or extract transform and load. So a data warehouse is like a common place where the data from different sources is brought together so that people can perform data science activities like reporting or data mining or statistical analysis and so on. So data from various sources is put in a centralized place which is known as a data warehouse. So that is also known as ETL. And in order to do this there can be data scientists can take help of some ETL tools. There are some existing tools that a data scientist can take help of like for example data stage or talent or Informatica. These are pretty good tools for performing these ETL activities and getting the data. The next stage now that you have the raw data into a data warehouse, you still probably are not in a position to straight away use this data for performing the data mining activities. So that is where data preparation comes into play and there are multiple reasons for that. One of them could be the data is dirty. There are some missing values and so on and so forth. So a lot of time is actually spent in this particular stage. So a data scientist spends a lot of time almost 60 to 70% of the time in this part of the project or the process which is data preparation. So there are again within this there can be multiple sub activities starting from let's say data cleaning you will probably have missing values the data there is some columns the values are missing or the values are incorrect uh there are null values and so on and so forth. So that is basically the data cleaning part of it. then you need to perform certain transformations like for example normalizing the data and so on right so or you could probably have to modify a categorical values into numerical values and so on and so forth. So these are transformational activities then we may have to handle outliers. So the data could be such that there are a few values which are way beyond the normal behavior of the data for whatever reason either people have keyed in wrong values or for some reason some of the values are completely out of range. So those are known as outliers. So there are certain ways of handling these outliers and detecting and handling these outliers. So this is a part of what is known as exploratory analysis. So you quickly explore the data to find out are there. So and you can use visual tools like plots and uh identify what are the outliers and see how we can get rid of the outliers and so on. Then the next part could be data integrity. Data integrity is to validate for example if there are some primary keys that all the primary keys are populated. there are some foreign keys then at least most of the foreign keys should be populated and otherwise when we are trying to query the data you may get wrong values and so on. So that is the data integrity part of it and then we have what is known as data reduction. Sometimes we may have duplicate values we may have columns that may be duplicated because they're coming from different sources. The same values are there and so on. So a lot of this can be done using what is known as data reduction and thereby you can reduce the size of the data drastically because very often this could be redundant data which can be removed and so on. So let's take a look at what are the various techniques that are used for data cleaning. So we need to ensure that the data is valid and it is consistent and uniform and accurate. So these are the various parameters that we need to ensure as a part of the data cleaning process. Now what are the techniques that that are used for data cleaning or uh so we will see what each of these are in this particular case and uh so what is the data set that we have? We have uh data about a bank and its customer details. So let's take an example and see how we go about cleaning the data. And in this particular example we're assuming we are using Python. So let's assume we loaded this data which is the raw file CSV. This is how the customer data looks like and um we will see for example we take a closer look at the geography column we will see that there are quite a few blank spaces. So how do we go about when we have some blank spaces or if it is a string value then we put a empty string here or we just use a space or empty string. If they are numerical values then we need to come up with a strategy. Uh for example we put the mean value. So wherever it is missing we find the mean for that particular column. So in this case let's assume we have credit score and we see that quite a few of these values are missing. So what do we do here? We find the mean for this column for all the existing values and we found that the mean is equal to 638.6. six. So we kind of write a piece of code to replace wherever there are blank values. N is basically like null and uh we just go ahead and say fill it with the mean value. So this is the piece of code we are writing to fill it. So all the blanks or all the null values get replaced with the mean value. Now one of the reasons for doing this is that very often if you have some such situation many of your statistical functions may not even work. So that's the reason you need to fill up these values or either get rid of these records or fill up these values with something meaningful. So this is one mechanism which is basically using a mean. There are a few others as we move forward. We can see what are the other ways. For example, we can also say that any missing value in a particular row if even one column the value is missing you just drop that particular row or delete all rows where even a single column has missing values. So that is one way of dealing. Now the problem here can be that if a lot of data has let's say one or two columns missing and uh we drop many such rows then overall you may lose out on let's say 60% of the data has some value or the other missing 60% of the rows then it may not be a good idea to delete all the rows like in that manner because then you're losing pretty much 60% of your data therefore your analysis won't be accurate. But if it is only five or 10% then this will work. Another way is only to drop values where or rather drop rows where all the columns are empty which makes sense because that means that record is of really no use because it has no information in it. So there can be some situations like that. So we can provide a condition saying that drop the records where all the columns are blank or not applicable. We can also specify some kind of a threshold. Let's say you have 10 or 20 columns in a row. You can specify that maybe five columns are blank or null then you drop that record. So again we need to take care that such a condition such a situation the amount of data that has been removed or excluded is not large. If it is like maybe 5% maximum 10% then it's okay. But by doing this if you're losing out on a large chunk of data then it may not be a good idea. You need to come up with something better. What else we need to do next is so the data preparation part is done. So now we get into the data mining part. So what exactly we do in data mining? Primarily we come up with ways to take meaningful decisions. So data mining will give us insights into the data what is existing there and then we can do additional stuff like maybe machine learning and so on to get perform advanced analytics and so on. So the one of the first steps we do is what is known as data discovery and uh which is basically like exploratory analysis. So we can use tools like Tableau for doing some of this. So let's just take a quick look at how we go about that. So Tableau is excellent data mining or actually more of a reporting or a BI tool and you can download a trial version of Tableau at tableau.com or there is also Tableau public which is free and you can actually use and play around. However, if you want to use it for enterprise purpose then is a commercial software. So you need to purchase license and you can then run some of the data mining activities. Let's say your data source your data is in some Excel sheet. So you can select the source as Microsoft Excel or any other format and the data will be brought into the Tableau environment and then it will show you what is known as dimensions and uh measures. So dimensions are all the descriptive columns. So and Tableau is intelligent enough to actually identify these dimensions and measures. So measures are the numerical values. So as you can see here uh customer ID, gender, geography these are all dimensions non-numerical values whereas age, balance, credit score and so on are numeric values. So they come under measures. So you've got your data into Tableau and then you want to let's say build a small model and you want to let's say solve a particular problem. So what is the problem statement? All right, let's say we want to analyze why customers are leaving the bank which is known as uh exit and we want to analyze and see if what are some of the factors for exiting the bank and we want to let's assume consider these uh three of them like let's say gender, credit card and geography these as a criteria and analyze if these are in any way impacting or have some bearing on the customer exiting or the customer exit behavior. Okay. So let's um use Tableau and very quickly we will be able to find out how these uh parameters are affecting. All right. So let's see. So this is our customer data. So from our Excel sheet we have data set about let's say 10,000 rows and we want to find out what is the criteria. Let's start with gender. Let's say we want to first use gender as a criteria. So Tableau really offers an easy drag and drop kind of a mechanism. So that makes it really really easy to perform this kind of analysis. So what we need to do is exited says whether the customer has exited or not. So it has a value of zero and one and then of course you have gender and so on. So we will take these two and simply drag and drop. Okay. So exited and then we will put gender. And if we drag and drop into the analysis side of of Tableau. All right. So here what we are doing is we are showing male female as the two different columns here and zero for people who did not exist and one for people who exited and that is colorcoded. So the blue color means people who did not exit and uh this yellow color means people who did exit. All right. So now if we pull the data here create like bar graphs this is how it would look. Uh so what is yellow? Let's go back. So yellow is uh who exited and uh for the male only 16.45% have exited and we can also draw a reference line that will help us or even provide aliases. So these are a lot of fancy stuff that is um provided by Tableau. You can create aliases and so that it looks good rather than basic labels and you can also add a reference line. So you add a reference line something like this. From here we can make out that on an average female customers exit more than the male customers. Right? So that is what we are seeing here on an average. So we have analyzed based on gender. We do see that there is some difference in the male and female behavior. Now let's take the next criteria which is the credit card. So let's see if having a credit card has any impact on the customer exit behavior. So just like before we drag and drop the credit card has credit card column if we drag and drop here and then we will see that there is pretty much no difference between people having credit card and not having credit card. 20.81% of people who have no credit card have exited and similarly 20.18% of people who have credit card have also exited. So the credit card is not having much of an impact. That's what this piece of analysis shows. Last we will basically go and check how the geography is impacting. So once again we can drag and drop geography column onto this side. And uh if we see here there are geographies like I think there are about three geographies like France, Germany and uh Spain and um we see that there is some kind of a impact with the geography as well. Okay. So what we derive from this is that the credit card is really we can ignore the credit card variable or feature from our analysis because that doesn't have any impact but gender and geography we can keep and do further analysis. Okay. All right. So what are some of the advantages of data mining? Bit more detailed analysis can help us in predicting the future trends and it also helps in identifying customer behavior patterns. Okay. So you can take informed decisions because the data is telling you or providing you with some insights and then you take a decision based on that. If there is any fraudulent activity, data mining will help in quickly identifying such a fraud as well and of course it will also help us in identifying the right algorithm for performing more advanced data mining activities like machine learning and so on. All right. So the next activity now that we have the data we have prepared the data and perform some data mining activity the next step is model building. Let's take a look at model building. So what is model building? If we want to perform a more detailed data mining activity like maybe perform some machine learning then you need to build a model. And how do you build a model? First thing is you need to select which algorithm you want to use to solve uh the problem on hand and also what kind of data that is available and so on and so forth. So you need to make a a choice of the algorithm and based on that you go ahead and create a model train the model and so on. Now machine learning is kind of at a very high level classified into supervised and unsupervised. So if we want to predict a continuous value could be a price or a temperature or or a height or a length or things like that. So those are continuous values and if you want to find some of those then you use techniques like regression, linear regression, simple linear regression, multiple linear regression and so on. So these are the algorithms. On the other hand, there will be situations or there may be situations where you need to perform unsupervised learning. In case of unsupervised learning, you don't have any historical labeled data so to learn from. So that is when you use unsupervised learning. And uh some of the algorithms in unsupervised learning are clustering. K means clustering is the most common algorithm used in unsupervised learning. And similarly in supervised learning if you want to perform some activity on categorical values like for example it is not measured but it is counted like you want to classify whether this image is a cat or a dog whether you want to classify whether this customer will buy the product or not or you want to classify whether this email is spam or not spam. So these are examples of categorical values and uh these are examples of classification. Then you have algorithms like logistic regression, K nearest neighbor or KNN and support vector machine. So these are some of the algorithms that are used in this case. And similarly in case of unsupervised learning, if you need to perform on categorical values, you have some algorithms like association analysis and hidden marco model. Okay. So in order to understand this better, let's take uh an example and uh take you through the whole process and then we will also see how the code can be written to perform this. Now let's take our example here where we want to perform a supervised learning which is basically we want to do a multilinear regression which means there are multiple independent variables and then we want to perform a linear regression to predict certain value. So in this particular example we have world happiness data. So this is a data about the happiness quotient of people from various countries and we are trying to predict and see whether our how our model will perform. So what is the question that we need to ask? First of all how to describe the data and then can we make a predictive model to calculate the happiness score. Right? So based on this we can then decide on what algorithm to use and what model to use and so on. So variables that are available or used in this model. This is a list of variables that are available. There is a happiness rank. I'll load the data and or I'll show you the data in a little bit so it becomes clear what are these. So there is what is known as a happiness rank. Happiness score which is happiness score is more like a absolute value whereas rank is what is the ranking and then which country we are talking about and within that country which region and what kind of economy and whether the family which family and health details and freedom trust generosity and so on and so forth. So there are multiple variables that are available to us and uh the specific details probably are not required and there can be um in another example the variables can be completely different. So we don't have to go into the details of what exactly these variables are but it's just enough to understand that we have a bunch of these variables and now we need to use either all or some of these variables and then which we also sometimes refer to as features and then we need to build our model and train our model. All right. So let's assume we will use Python in order to perform this analysis or perform this machine learning activity. And I will actually show you in our lab in in a little bit this whole thing. We will run the live code. But quickly I will run you through the slides and then we will go into the lab. So what are we doing here? First thing we need to do is import a bunch of libraries in Python which are required to perform our analysis. Most of these are for manipulating the data, the preparing the data and then scikitlearn or skarn is the library which we will use actually for this particular machine learning activity which is linear regression. So we have numpy, we have pandas and so on and so forth. All these libraries are imported and then we load our data and the data is in the form of a CSV file and there are different files for each year. So we have data for 2015, 16 and 17. And uh so we will load this data and then combine them, concatenate them to prepare a single data frame. And uh here we are making an assumption that you are familiar with Python. So it becomes easier if you are familiar with Python programming language or at least some programming language so that you can at least understand by looking at the code. So we are reading the file each of these files for each year and this is basically we are creating a list of all the names of the columns we will be using later on you will see in the code. So we have loaded 2015 then 2016 and then also 2017. So we have created um three data frames and then we concatenate all these three data frames. This is what we are doing here. Then we identify which of these columns are required. Which for example some of the categorical values do we really need? We probably don't. Then we drop those columns so that we don't unnecessarily use all the columns and make the computation complicated. We can then create some plots using plotly library and it has some powerful features including creation or creation of maps and so on. just to understand the pattern the happiness quotient or how the happiness is across all the countries. So it's a nice visualization we can see each of these countries how they are in terms of their happiness score. This is the legend here. So the lighter colored countries have lower ranking and so these are the lower ranking ones and these are higher ranking which means that the ones with these dark colors are the happiest ones. So as you can see here Australia and maybe beside uh US and so on are the happiest ones. Okay. The other thing that we need to do is the correlation between the happiness score and happiness rank. We can find a correlation using a scatter plot and we find that yes they are kind of inversely proportion which is obvious. So if the score is high happiness score is high then they are ranked number one for example highest is scored as number one. So that's the idea behind this. So the happiness score given here and the happiness rank is actually given here. So they are inversely proportional because the higher the score the the absolute value of the rank will be lower. Right? Number one has the highest value of the score and so on. So they are inversely correlated but there is a strong what this graph shows is that there is a strong correlation between happiness rank and happiness score. And then we do some more plots to visualize this. we determined that probably rank and score are pretty much conveying the same message. So we don't need both of them. So we will kind of drop one of them and uh that is what we are doing here. So we drop the happiness rank and similarly. So this is one example of how we can remove some columns which are not adding value. So we will see in the code as well how that works. Moving on, this is a correlation between pretty much each of the columns with the other columns. So this is a correlation you can plot using plot function and uh we will see here that for example happiness score and happiness score are correlated strongest correlation right because every variable will be highly correlated to itself so that's the reason so the darker the color is the higher the correlation and as so the and correlation in numerical terms goes from 0 to one so one is the highest value and it can only be between 0 and one correlation between two variables can be only have a value between 0 and one. So the numerical value can go from 0 to one and one here is dark color and zero is kind of dark but it is blue color. From red it goes down. The dark blue color indicates pretty much no correlation. So the from this heat map we see that happiness and economy and family are probably also health probably are the most correlated and then it keeps decreasing after freedom kind of keeps decreasing and coming to pretty much uh zero. All right. So that is a correlation graph and then we can probably use this to find out which are the columns that need to be dropped which do not have very high correlation and uh we take only those columns that we will need. So this is the code for dropping some of the columns. Once we have prepared the data when we have the required columns then we use scikitlearn to actually split the data. First of all, this is a normal machine learning process. You need to split the data into training and test data set. In this case, we are splitting into 80/20. So 80 is the training data set and 20 is the test data set. So that's what we are doing here. So we use train test split method or function. So you have all your training data in x train the labels in y train. Similarly x test has the test data the inputs whereas the labels are in y test. So that's how and this value whether it is 8020 or 50/50 that is all individual preference. So in our case we are using 8020. All right. And uh then the next is to create a linear regression instance. So this is what we are doing. We're creating an instance of linear regression and then we train the model using the fit function. And uh we are passing x and y which is the x value and the label data regular input and the label data label information. Then we do the test we run the or we perform the evaluation on the test data set. So this is what we are doing with the test data set and then we will evaluate how accurate the model is and using the scikit learn functionality itself. We can also see what are the various parameters and what are the various coefficients because in linear regression you will get like a equation of like a straight line y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 those beta 1 beta 2 beta 3 are known as the coefficients and beta 0 is the intercept. After the training you can actually get these information of the model what is the intercept value what are the coefficients and so on by using these functions. So let's take quickly go into the lab and take a look at our code. Okay. So this is my lab. This is my Jupyter notebook where the code I have the actual code and I will take you through this code to run this linear regression on the world happiness data. So we will import a bunch of libraries numpy pandas plot plotly and so on also. Yeah, scikit learn that's also very important. So that's the first step. Then I will import my data and uh the data is in three parts. There are three files one for each year 2015, 2016 and 2017. And it is a CSV file. So I've imported my data. Let's take a look at the data quickly glance at data. So this is how it looks. We have the country, region, happiness rank and then happiness score. there are some standard errors and then what is the per capita family and so on. So and then we will keep going. We will create a list of all these column names we will be using later. So for now just we I will run this code. No need of major explanation at this point. We know that some of these columns probably are not required. So you can use this drop functionality to remove some of the columns which we don't need like for example region and standard error will not be contributing to our model. So we will basically drop those values out here. So we use the drop and then we created a vector with these names column names. That's what we are passing here. Instead of giving the names of the columns here we can pass a vector. So that's what we are doing. So this will drop from our data frame. It will remove region and standard error these two columns. Then the next step we will read the data for 2016 and also 2017. And then we will concatenate this data. So let's do that. So we have now data frame called happiness which is a concatenation of both all the three files. Let's take a quick look at the data now. So most of the unwanted columns have been removed and you have all the data in one place for all the three years. And this is how the data looks. And if you want to take a look at the summary of the columns, you can say describe and uh you will get this information. For example, for each of the columns, what is the count? What's what is the mean value, standard deviation, especially the numeric values, okay? Not the categorical values. So this is a quick way to see how the data is and uh initial little bit of exploratory analysis can be done here. So what is the maximum value? What's the minimum value and so on for each of the columns. All right. So then we go ahead and create some visualizations using plotly. So let us go and build a plot. So if we see here now this is the relation correlation between happiness rank and happiness score. This is what we have seen in the slides as well. We can see that there is a tight correlation between them. Only thing is it is inverse correlation but otherwise they are very tightly correlated which also says that they both probably provide the same information. So there is no not much of value add. So we'll go ahead and drop the happiness rank as well from our columns. So that's what we're doing here. And now we can do the creation of the correlation heat [snorts] map. Let us plot the correlation heat map to see how each of these columns is correlated to the others and we as we have seen in the slides. This is how it looks. So happiness score is very highly correlated. So this is the legend we have seen in the slide as well. So blue color indicates pretty much zero or very low correlation. Deep red color indicates very high correlation. And the value correlation is a numeric value and the value goes from 0 to one. If the two items or two features or columns are highly correlated then they will be as close to one as possible and two columns that are not at all correlated will be as close to zero as possible. So that's how it is. For example here happiness score and happiness score every column or every feature will be highly correlated to itself. So it is like between them there will be correlation value will be one. So that's why we see deep red color. But then others are for example with higher values are economy and then health and then maybe family and freedom. So these are generosity and trust are not very highly correlated to happiness score. So that is uh one quick exploratory analysis we can do and uh therefore we can drop the country and the happiness rank because they also again don't have any major impact on the analysis on our analysis. So now we have prepared our data. There was no need to clean the data because the data was clean. But if there were some missing values and so on as we have discussed in the slides, we would have had to perform some of the data cleaning activities as well. But in this case, the data was clean. All we needed to do was just the preparation part. So we removed some unwanted columns and we did some exploratory data analysis. Now we are ready to perform the machine learning activity. So we use scikitlearn for doing the machine learning. Scikitlearn is Python library that is available for performing our uh machine learning. Once again we will import some of these libraries like pandas and numpy and also scikitlearn. First step we will do is split the data in uh 2080 format. So you have all the test data which is 20% of the data is test data and 80% is your training data. So this test size indicates how much of it is in the what is the size of the test data. the remaining which is here we are saying 02 therefore that means training is 08 so training data is 80%. All right so we have executed that split the data and now we create an instance of the linear regression model so lm is our linear regression model and we pass x and y the training data set and call the function fit so that the model gets trained. So now once that is done, training is done, training is completed and now what we have to do is we need to predict the values for the test data. So the next step is using so you see here fit will basically run the training method. Predict will actually predict the values. So we are passing the input values which is the independent variables and we are asking for the values of the dependent variable which is which we are capturing in y prime and we use the predict method here lm.predict. So this will give us all the predicted y values and remember we already have y test has the actual values which are the labels so that we can use these two to compare and find out how much of it is error. So that's what we are doing here. We are trying to find the difference between the predicted value and the actual value. Y test is the actual value for the test data and Y predict is the predicted value. We just found out the predicted value. So we will run that and we can do a quick check as to how the data looks. How is the difference? So in some cases it is positive, some cases it is negative. But in most of the cases I think the difference is very small. This is exponential to the power of 0 - 04 and so on. So looks like our model has performed reasonably well. We can now check some of the parameters of our model like the intercept and the coefficients. So that's what we are doing here. So these are the coefficients of the various parameters that we or the coefficients of the various independent variables. Okay. So these are the values. Then we can quickly go ahead and list them down as well against the corresponding independent variables. So the coefficients against the corresponding independent variable. So 1.0051 051 is the coefficient for economy. N9983 is for family, coefficient for family and health and so on and so forth. Right? So that's what this is showing. Now we can use the functionality readily available functionality of scikitlearn and then plot that to find some of the parameters which determine the accuracy of this model like for example what is the mean square error and so on. So that's what we are doing here. So let's just go ahead and run this. So you can see here that the root mean square error is pretty low which is a good sign and uh which is one of the measures of u how well our model is performing. We can do one more quick plot to just see how the actual values and the predicted values are looking. And once again you can see that as we have seen from the root mean square error root mean square error is very very low. That means that the actual values and the predicted values are pretty much matching up almost matching up. And this plot also shows the same. So this line is going through the predicted values and the actual values and the difference is very very low. So again this is actual data. This is one example where the the accuracy is high and the predicted values are pretty much matching with the actual values. But in real life you may find that these values are slightly more scattered and you may get the error value can be relatively on the higher side. The root mean square error. Okay. So this was a good uh quick example of uh the code to perform data science activity or machine learning or data mining activity. In this case we did what is known as linear regression. So let's go back to our slides and see what else is there. So we saw this these are the coefficients of each of the features in our code and uh we have seen the root mean square error as well and uh with we can take a few hundred countries certain values and actually predict to see if how the model is performing and I think we have done this as well and in this case as we have seen pretty much the predicted values and the actual values are pretty much matching which means our model is almost 100% accurate as I mentioned it real life it may not be the case but in this particular case we have got a pretty good model which is very good also subsequently we can assume that this is how the equation in linear regression the model is nothing but an equation like y is equal to beta 0 plus beta 1 x1 plus beta 2 x2 plus beta 3 x3 and so on. So this is what we are showing here. So this is our intercept which is beta 0 and then we have beta 1 into economy value, beta 2 into the family value, beta 3 into health value and so on. So that is what is shown here. Okay. So I think the next step once we have the results from the data mining or machine learning activity the next step is to communicate these results to the appropriate stakeholders. So that is what we will see here now. So how do we communicate? Usually you take these results and then either prepare a presentation or put it in a document and then show them these actionable results orable insights and uh you need to find out who are your target audience and uh put all the results in context and uh maybe if there was a problem statement you need to put this results in the context of the problem statement. what was our initial goal that we wanted to achieve. So that we need to communicate here based on you remember we started off with what is the question and what is the data and so on and then what is the answer. So we we need to put the results and then what is the methodology that we have used all that has to be put and clearly communicated in business terms so that the people understand very well from a business perspective. So once the model building is done, once the results are published and communicated, the last part is maintenance of this model. Now very often what can happen is the model may have to be subsequently updated or modified because of multiple reasons. Either the the data has changed, the way the data comes has changed or the process has changed or for whatever reason the accuracy may keep changing. Once you have a trained model the for example we got a very high accuracy but then over a period of time there can be various factors which can cause that. So from time to time we need to check whether the model is performing well or not. The accuracy needs to be tested once in a while and if required you may have to rebuild or retrain the model. So you do the assessment you you see if it needs any tweaks or changes and then if it is required you need to probably retrain the model with the latest data that you have and then you deploy it. You build the model, train it and then you deploy it. So that is like the maintenance cycle that you may have to take the model. Data analyst versus data engineer versus data scientist. Which one to choose? This is one of the most popular questions asked by learners looking for a career in data and analytics. I'm sure you two would have come across these job roles in the ever growing data science landscape. Though they all deal with data, these jobs are not the same. There are significant differences between what a data analyst, data engineer, and a data scientist does. We will look at these job roles and the differences in detail. First, let's look at some data analytics and data science trends. The analytics and data science market is thriving. Data analytics, data engineering, and data science are the key trends in today's exhilarating market. As per statist.com, the global big data analytics market revenue will grow at a caggr of 30% with revenue reaching over 68 billion US by 2025. According to Technavio, the enterprise data management market is expected to increase by 64.08 billion US by 2025 as per markets and markets.com. The big data market size is projected to grow from 162.6 billion US in 2021 to $273.4 billion US in 2026. Now another report from research drive says that the data science platform market is estimated to reach 224.3 billion US by 2026. So with so much data available and companies making huge investments to drive business insights the job opportunities for data analysts, data engineers and data scientists are going to increase in 2022 and over the coming years. Now let's learn the major differences between data analyst versus data engineer versus data scientist. So who are they? A data analyst analyzes and interprets vast volumes of data in order to extract meaningful information out of it. They find solutions to a business problem and make critical business decisions. The insights provided by data analysts are important to companies that want to understand the needs of their end customers. But talking about who a data engineer is, a data engineer on the other hand builds infrastructure and scalable pipelines to manage the flow of data and prepare it for analysis. So basically they optimize the systems that enable data analysts and data scientists to perform their job efficiently. Data scientists are professionals who analyze and visualize existing data and use algorithms to build predictive models for making future decisions. They also engage with business leaders to understand their needs and present complex findings. With that, let's look at the primary roles and responsibilities of these three job roles. Data analysts are responsible to collect, clean, store and process data. They discover hidden patterns from data by performing exploratory data analysis and visualize data by creating charts and graphs. Acquiring data from primary and secondary sources is one of their key tasks. They build reports and dashboards and also maintain databases. Now talking about the roles and responsibilities of a data engineer. A data engineer performs data acquisition, the design, build and test data as well to develop and maintain data architecture. Data engineers are tasked with testing, integrating, managing and optimizing data from a variety of sources. So they integrate data into existing data pipelines, prepare data for modeling and perform various ETL operations. Now talking about the roles and responsibilities of a data scientist. So data scientists develop machine learning models to identify trends in data for making decisions. They develop hypothesis and use their knowledge of statistics, data visualization and machine learning to forecast the future for the business. Data scientists visualize data and use storytelling techniques and also write programs to automate data collection and processing. Now move on to the skills possessed by data analysts, data engineers and data scientists. To become a data analyst, you need to have good hands-on experience with writing SQL queries. You should have excellent Microsoft Excel skills for analyzing data. Data analysts are also good at programming and they need to know how to visualize data, solve business problems, and possess domain knowledge. Data engineers should have a solid understanding of SQL, MongoDB, and programming. They need to have a good command of data architecture, scripting, data warehousing and ETL. Data engineers are also good at Hadoop based analytics. Now talking about the skills for a data scientist. So a data scientist should have experience with programming in Python and R. They should have a very good understanding of mathematics and statistics as well. Data scientists need to possess analytical thinking and data visualization skills as well. Machine learning, deep learning and decision-m are other critical skills