Machine Learning Full Course 2026 [FREE] | Machine Learning Tutorial For Beginners | Simplilearn

Key Takeaways

Welcome to the machine learning course by simply learn. Machine learning is one of the most important skills in today's job market because companies want professional who can learn from data and make smart predictions. In this course, you will start from the basic and slowly move towards building real machine learning models. You will not only just learn definitions but also understand how algorithms work, why model fails and how they are used in real projects. By the end of this course, you will have a strong foundation in machine learning that can help you grow into roles like data analyst, machine learning engineer or even data scientist. So let's talk about the agenda now. First, we will understand what machine learning is and how it is different from traditional programming. Second, we will learn core concepts like train, test, split, overfitting and how models learn from data. Third, we will understand linear regression and a math behind it in simple way. Next, we'll learn how to evaluate models using matrices like R squar or adjusted R squar. Then we will prepare the data by handling categorical value and checking model assumptions. After that, we will build and deploy a complete machine learning project using streamllet. Finally, we'll learn classification using logistic regression and evaluate results using a confusion matrix. So before we move on, here's a quick information for you. If you are serious about building a strong career in AI and machine learning, the professional certificate in AI and machine learning by simply learn in partnership with Perio University is a great option to consider. This six-month live online program is designed for working professional and focuses on real in- demand skills like machine learning, deep learning, generative AI, chat, GPT, LLMS and agentic AI. You will learn through live interactive classes, work on 15 plus hands-on projects and use industry tools while getting guidance from industry experts and IBM masterclass. On completion, you'll also receive a certificate from Simply Learn in collaboration with Purdue University along with career support to help you move into high growth AI roles. So, hurry up and enroll now and you can find the course link in the description box below and in the pin comments. Before we move on, here's a short quiz to test your understanding. What is machine learning mainly used for? Writing fixed rules, learning patterns from data, designing website, or storing files. So friends uh let's start with it. Let me give the machine learning. Even before starting machine learning what we need to understand is what is ideally the need of machine learning. So friends, we all have heard that data is the new oil today. So when I say data is the new oil, what do we exactly mean? If I come back to my days of doing my engineering somewhere around the year uh 2005 to 2009 when I passing 2009 when I was doing my bachelor's of engineering so I would say that during our time when I started I was using the 1.4 and 4 MB floppy disc. So that was in itself a big achievement. Then a 4 MB floppy disc came into picture. Then during our time only a compact disc came which could have saved data up to 100 MB. So we were very happy. During our time only a special CD came which is called as data CD which can save data up to 700 MB specifically for saving data not for uh so basically saving movies or anything that was good. Then came the DVD. Then during our time only came the Blu-ray because if someone is a millennial maybe he might not be knowing it. So Blu-ray CD came and then during our I actually experienced this pen drive came very very small size and then we started having pen drives of bigger size and further bigger size and then it all is going fast and fast from here only. Earlier computers were as big as a room. Now we have powerful computers with GPU all into a laptop. So everyone's standard of living or I would say the minimum things that we need to survive that baseline has increased. There was a time where in every home there used to be only one working person feeding the whole family. Today many people have to work. That has also changed. Plus in every home there was one landline telephone. Mobile in itself was a big thing. Today every person has a mobile phone. So that has become a need. No one can survive without a phone. That's a fact. What I want to say is the baseline standard of living irrespective of whether you are poor, middle class or high class, the baseline standard of living has increased. We all have in our house a flat TV. Gone are those days where you had TVs which have a box behind which is projecting. That's gone. We all have flat TV. Everyone has a mobile phone, right? And I think the cost of everyone's mobile phone is somewhere between on an average 7,000 to 20,000 on an average. And every person in the family has it. Even if you are poor, at least there will be one bike in your house. And if you are rich then you have a bike and a car right so these all things have become bare minimum now so basically we are progressing right that's the overall story now now the thing which I wanted to say is we were talking about data So data is what we call today as the new oil. In fact simply learn somewhere in the year just just before co the advertisement used to come whenever I used to open YouTube. Have you heard data is the new oil and blah blah blah blah that advertise used to continue after that and then after that I used to skip that ad. But data is the new oil. That was one of the very important thing. So data is the new oil doesn't really mean that data is the oil. But data is as precious as oil. I remember when I was small, right? We used to live in a small house because uh my father actually came from zero to infinity. So what I wanted to say over here is uh so it was a building something like this where in the center we had a staircase. There were four floors in the building. So four houses on the ground floor here, four houses here. First floor four houses. Second floor four houses. Third floor four houses. Fourth floor four houses. And the last floor. That was how it was. And I remember we used to stay here, right? So people used to come to take surveys with a pad having papers and a pen in their hand. They used to ask questions like which shampoo do you use? Which soap do you use? What is your average family income? Do you have a bike? Do you have a car? How many people are working in your family? And such questions which are usually taken in a survey. What do you like? And then after that if you give that survey they used to take around 10 minutes to sometimes 15 minutes of the time. If you take that survey, they used to give a free complimentary bottle of their product. Could be a shampoo or could be a small new taster of a new milk product which they are launching or anything. Many people used to come like that. But what were we do? But what were all of them doing? Data collection. I don't think so. Today people like this exist anymore because all the data that we are generating is coming from here. Right? Am I correct? Earlier the data was collected for entire family that to after so many efforts today every person is generating data our session is getting recorded we are also generating data right so if I am going on to the Instagram right so The kind of reels which will be recommended to me will also be purely based on my watch history etc. Or if I go on to my YouTube again the kind of res which are recommended to me will be on the type of my watch history. Oh what happened over here? That's sad. So essentially because they have my data, they have studied that is >> this baby elephant lost. >> How is that possible? So what I wanted to say is every one of us is generating data and that is exactly where machine learning right. So I say machine learning is a powerful concept as well as framework for making the best out of it. So today we say that we are living in what we called as the age of information or the age of data. Next thing over here let me ask you a question. Who is the most advanced and intelligent on the planet at the moment? Who is the most advanced and intelligent on the planet at the moment? What will be your question? What will be your answer? So can I tell to my machine? You know does if machine is smart then let's consider TV. Does a TV automatically call you and say that hey your favorite program Taraka Jasma is coming in 15 minutes please come and watch it. Does it say then? How is that intelligent? How is it the most intelligent most advanced? Elon Musk is just one person. Uh so again when you say Elon Musk I assume that it's going to be humans. Uh a chat GP2 can't do anything it can just generate text just because of that we can't really call it smart someone which is just generating the text. For example, if I go to chat GBD and say that uh book me my flight tickets for tomorrow from Mumbai to Bangalore for 11:00 a.m. Can GBT do that for me? Can it do can it book my flight tickets if I give it a budget? Tell me can it do or it cannot do? Because someone said that it is very intelligent. It has information but it can't perform actions. Uh we can say that's exactly where agentic AI is coming into picture. But again it is not the most smart as of now at least that's what one thing is for sure. So if if you say that is what is called as human then obviously the question comes to our mind is why do we really need machine learning? If this is what is there then why do we really need machine learning and the answer is you cannot although you say humans are very smart but humans cannot yes very nicely said sesh humans cannot make decision at a scale for example if I say I have a data set of one lakh rows and if I say study this one lakh rows and take a decision. You cannot do that. You cannot remember so many things. That is exactly where machines can help. So why do we need machine learning? We needed to make datadriven decisions at a scale. That's something which is very very important. So as a human whenever we have to make a decision we use what is called as power of reasoning or intuition. So let me ask you a very simple question. Since you said humans are most intelligent, you have a friend of yours who you know since past 10 years. He comes to you and asks you, will you lend me 10,000 rupees? What is your answer? A friend of you comes to you and asks for this. What will be your answer? You know him since past 10 years. Will you lend him or not? It's a human decision. You don't need a machine for that. I actually did the same thing. But I never got it back. Yes, I never got it back. So that person was in my college group only 2005 to 2009. It took the money and he never written me. Working now as a uh manager or some team leader in some company but he did not return me and now he drinks a lot and he has totally changed. He has divorced as well. So people change over time is what I wanted to say. So I was saying people change over the time same right? Yes. So we really can't actually rely on this and you know more than whether he returns or not I was I'm more interested in understanding how we took the decision. How did you all take the decision? So I'm sure you might have looked at the past history with this friend in the sense you may be like let's say in the past how many years 10 years you interacted with your friends. Let's say in the past 10 years you interacted with your friend total 100 times. So basically you met your friend 100 times. So when he comes and asks you can you lend me 10,000 rupees. Let's say all of a sudden I'm your friend. I come to you and I say that sheer please give me 10,000 rupees. I am in need. So will shaker remember all the 100 interactions which he had with me instantly when I ask him this question. Is it possible for our brain to remember everything? Impossible. Right? Maybe maybe yes maybe out of this 10 shaker or in our case human may be able to remember only the most prominent or we can even say the most recent 10 interactions. So what will Shakhar say that out of 10 times my friend Dian was seven times good with me three times he was not that helpful but overall Dian is a good guy so I will lend him money and that's how shaker takes a decision but imagine all of this data if it is saved in a machine let's let's assume all of this 100 interactions are saved in a Microsoft Excel file where interaction and then it was a good interaction or not. Interaction was that good or not? It was good, it was good, not good dot dot dot dot dot. Now when machine has to take a decision machine is like okay Dan this guy was 79 times good with you and 21 times he was not that good with you. So according to me you take a decision. Yes. And that is how machine also takes a decision. So both of your decisions are correct. But whose decision in this case is more reliable? Whose decision is more reliable? Both of you are correct. That's good. Machine. Yes. Machine. At least for now we understand that the decision of machine is more reliable. But there is a problem. Is this machine learning? No, this is not machine learning. So here I think this this decision which is there it is just easily possible by applying a group by on the Excel file or maybe filtering out the yes and the filtering out of no this has been something since a long time since Microsoft Excel came is this machine learning no plus here let's say instead of 100 if you had 50,000 interactions then every time for taking a decision. You have to refer this Excel file. Find out how many positive, how many negative and then take a decision. That's not machine learning. This is like data analysis, EDA, what you were doing. So obviously we still don't understand what is machine learning. Well, let me ask you if you go to a doctor then so let's say for a person to become a doctor in five years of his time he has to study so many books. For a person to become a doctor let's say he has to study so many books. So now when Shakhar is not feeling well why Shikhar Dan I'm not feeling well so let me talk about me only. When Daran is not feeling well, he goes to the doctor who has actually studied this right. So I go to my family doctor and I say that hey this is the problem. What what should I do? So if I remember the Microsoft Excel file example where we had to look into this and find out how many good and how many bad were there and we were taking a decision. So when I go and tell to the doctor that I am not well does this guy as well open each and every book or whichever book is relevant and then says that okay Dan you are having throat and for that I think you should have this this medicine does it happen like that? No. So what exactly how does he take a decision? How does he take a decision then? What do you think? How a human mind is taking decision? Experience. So you mean to say books are not important. The knowledge that is studied knowledge and experience processing the data which is already there in the brain memory. Awesome. That is exactly where we are coming. So we can actually say it we can actually say that if I find out the weight of all these books if it is coming to around 250 kilogram the doctor has studied 250 kilogram of data. I say the doctor has studied 250 kilg of the data but all of this data is saved into his brain which is something like this right that's how a human brain is. Can someone tell me what is the weight of a human brain? What's the weight of a human brain? Google it and quickly tell me kg insane it's around 1.3 to 1.4 4 kg. That's it. That's perfectly said. So that means that means 250 kilogram of data is saved into 1.3 kilogram of data or brain. And whenever I go to the doctor, the doctor is taking the decision on this 1.3 kilogram which has knowledge of his books, right? as you said knowledge of his books plus as you said his experience plus as you said his logic and many other things. Now that is what we call as machine learning for now for now because the doctor didn't save all this data. He learned the information from this data. We all know there is a difference between data and information. He learned the information and it is being used for taking a decision. It is being used for taking a decision. This is something very very important. Very very important. So when you say machine learning is creating a machine learning model, the model is not storing the data. I am repeating it. It is not storing the data but it is storing the feature learning that's something very important. So you will see usually the size of a machine learning model even if you ask it to make it even if you say that hey this is the 14 GB of data could be a Microsoft Excel file I want you to study that the size of a machine learning model is between few KBs to maximum the highest around 700 MB. So you can understand this is how it is going to take a decision. It is not referring to the file. So that also means just like when a doctor has to take a decision when I go to him what kind of decision am I having and what he should recommend me I can comfortably say that this doctor does not require this he does not require this he can take decision only on the basis of brain in The same way when you have to now predict whether your friend will lend money to you or not based on the past data, you don't require the past data. You only require what is called as a model. Are you all able to understand? So it's all about learning the features right? Yes. So now I am going ahead. So now I'm very soon maybe in uh another 5 minutes I will be starting with actually what is machine learning. So I'm going to Google now we need to spend at least at least 30 minutes on this. At least this single image. Yes, you heard it right. 30 minutes on this single image. advise this uh consider a very simple problem what used to happen so far. So we will talk with respect to traditional programming. So let us say you have to create a very simple application to perform addition of two numbers. Then you had to write a program. Who will write a program? A software developer will end up writing a program. Let's say you want to perform the addition of two numbers then write a program add py and then that program when executed on a computer you might be getting an executable file and then you ask the program because you are written a program that can perform the addition. you ask it 2 + 3. So this program takes this performs 2 + 3's and gives the output as five or maybe if you ask it what is 1.1 + 2.2 this will end up answering 3.3. So what I want to tell you in this scenario the output is deterministic in nature. What I mean by deterministic is 2 + 3 can be this. It's almost fine. Huh? So can I say 2 + 3 is equal to 5.01? Can I say 2 + 3 is equal to 4.99999? No. So you mean to say the answer has to be five and only five. It has to be deterministic. Can I say 1.1 + 2.2 can be 3.3001. Will you agree to this answer? That's exactly what I am saying. So let's say consider a very traditional application not a software program like addition. py. But let's say you created a bank website. So let's say you created a perfect website for a bank like HDFC. So as a customer, the customer said that I have 1 lak rupees to invest and I'm going to invest it for 3 years. The rate of interest which is being offered is let's say 5.12%. So after 3 years I am entitled to get 1.4 lak rupees. As a customer will you agree if the bank is paying you instead of 1.4 lak rupees 1 lakh uh 39,992 rupees instead of 1.40 the bank is paying this. So it's around 8 rupees less. Will you agree? No. You'll be like no. As per these calculations, I am entitled to get this much. I want this much only. It's not coming out to that. So essentially the output in the earlier programs was deterministic. But machine learning is different. And how is it different? That's exactly what we are going to understand. Now did you all understand what is traditional programming with the help of the addition and the bank example then I can go ahead to define machine learning then everyone is clear with what is traditional programming. So let's say consider I am your Python teacher and I completely taught you Python and I say that guys tomorrow is your exam on Python. to study and come and then you come to me and say uh syllabus and all things are very high. Can you help us with some sample questions and because I have been taking this subject since a very long time I do have a lot of questions already ready with me. Let's assume I have 80 questions and the best part is I also have their sample answers. What I say is since you come requesting for some sample questions, I am like okay no problem. Tomorrow I have to take their exam out of this 100 questions let me think of giving them around 80 questions to study. And this 20 questions as a teacher I will keep it separately so that I can use it for asking them in the examinations. So now I give you these 80 questions and also there are 80 answers but this is what is not shared with you. I'm not giving you this only this and this is shared. So what you do? You go home and then you study. And what do you exactly mean by study? You understand the relationship between this question, this answer, this question, this answer, this question, this answer, so on and so forth. You understand the relationship between every question and every answer. And where is it saved? It is saved into your brain. This entire information is saved into your brain. So what I do is tomorrow when you come for the examination. These are the questions which you have never seen as a teacher. I give this questions to you in the examination because you had studied this. you answer these 20 questions. Let us assume that these are the 20 answers that you gave. So I can say these are your predictions and these are your actual answers. As a teacher, I will compare the 20 answers which are actual and the 21s that you predicted. I am going to compare this and calculate how much good you are on this data. And that's exactly what I call as the accuracy on the test. And that's how I evaluate you. I could have also done one more thing because you studied this 80 questions and 80 answers. I might say that okay let me ask them the same 80 questions. So obviously you are expected to give 80 answers. So whatever you studied the same thing I asked and these are the predictions and these are the answers. Again I can compare this two because these are your predictions. These are the actual or the model answers. I can compare and find out what we call as training accuracy. Are you understanding this figure? What we did in this so far? This is kind of very very important for machine learning, right? Yes. So I repeat, we had I had 100 questions. I divided into 8020 where 80 questions and answers I ask you to study. And uh later on when your study is over and it's saved into the braille, I ask you this 20 questions which you didn't study and you answered on those 20. These 20 I compare with the actual ones and I find out how good you are on unseen something which you never saw. And then I ask you the same 80 questions. you give that answers and I see the actual ones I compare them and I f calculate your training accuracy. So in machine learning also we end up doing something same. So here this brain is nothing but called as a model. Now see what I mean to say is uh in uh whenever you talk about this predictions you don't actually mug up the data you are storing the features how many of you when I say that go and understand the relationship when I say go and understand the relationship in between the question and the answer. How many of you literally ratta mug up the things? How many of you mug up the things? Question answer question answer question answer. Do you do that? I'm exactly talking about like the chhatur of three idiots. If you ask him what is machine, how did he answer? Do you learn it that way so that tomorrow when I ask you what is machine and you start speaking machine is this this this you don't in fact in fact if I just randomly go and go on to let's say even if I ask you to let's say even if I ask you to study these two paragraphs let's say I give you 10 minutes of time and I say that study this and then I'll be asking you to rewrite it on a plain piece of paper. Do you think you will be able to actually write down each and every word as it is? I'm sure the answer is no. But if I say would you be able to reproduce the same concept without writing once you study the answer would be yes because you'll be writing those all things in your own words and that is exactly what is happening over here when you are understanding the relationship between the question and the answer and you say it is saved into your brain your brain is not actually mugging up the data but it is saving the pattern. Now here coming on to this figure what I want to say is now this time the 80 questions and the 80 answers that you study this is the input and in return what you get is nothing but a program or the brain or the model. So if you now remember earlier the program was written by the software developer but now here the program is actually written down by the computer and it's not actually a program it's a model. So earlier you see data and program was on the left side and results were on the right hand side but now you have data and results on the left hand side and the program on the right hand side. So you don't need a software developer to write the program but rather here your machine is the one who is actually going to know or who is actually going to do what you call as the feature learning and you get this but you cannot really trust the output of a machine. The machine may be right the machine may be wrong. The machine says yes dul I have done studying but how do you trust that machine? The only way you can trust that machine is by asking it questions. Tomorrow you come and say when I ask you to study these questions tomorrow let's say Raj comes to me and says that yes dash I am done with my study. Huh? Now Raj says that I'm done with my study. Everything is saved in my brain. Should I trust him? He might not have studied. He might have studied. How do I trust him? How do I really know whether he has studied or not? Unless and until I evaluate that's exactly how right from junior KG till whatever we are today we have come right. So junior to senior, senior to first standard, second standard, so on and so forth, 10th standard, so on and so forth, 12th standard, then medical engineering. That's how every time we have given an exam, the teacher is not like okay Raj is a very good boy. So Raj is going from first standard to second directly. No, irrespective of whether he's good or bad, he has to go through agniparika. Examination is important. He might say he has studied but what is the guarantee that he studied? I will take his exam. If he passes in the exam then I allow him to go to the next standard. So in the same way the computer or the machine may say that yes I have understood the relationship between the 80 questions and the 80 answers but that could be wrong understanding as well. So unless and till you validate it that's going to be a challenge always always remember it and that's why here as well we are doing the same thing in machine learning. So machine learning is about this single figure that you see on the screen. You give some data to the machine to study. It understands the relationship. Once the relationship is understood, you ask it something which it has not studied and it will predict. This prediction you compare with the actual and based on that you calculate the accuracy and confirm how good or bad is the model. Right? So that is essentially what is machine learning. But there are some challenges as well over here which I have to now discuss. Technically they are called as overfitting and underfitting. So when you say that okay Raj has studied well how do you actually quantify whether he really studied well or not I was saying over here now we are good to get into this understanding so huh I remember we're talking about overfitting and underfitting so what did we do over here see here we asked the we as I asked you study with 80 questions and 80 answers and then after this study is over you said the data is saved in my brain then I confirmed whether you are really understanding the things or not by asking you this exam questions you answered for this exam I compared it and I found out the test accuracy in the same way I said that let me ask the same questions that you studied you answered it again and I compared it and I calculated what we call as the train accuracy after this. So now the question is now that we have calculated it the question is what is a good model? A good machine learning model is the one which satisfies the below two conditions. Test accuracy must be greater than equal to 80%. And the second one is the train accuracy minus test accuracy difference must be less than equal to 5%. then and then we say it's a good model. Now this is something which is very very important to be noted by everyone. If the difference in between the train and test accuracy is less than 5%. And the test accuracy is at least 80%. Then you call it as a good machine learning model. Here we say let's consider a very simple scenario. I'll keep it open. And let's say we have the train accuracy. We have the test accuracy and this is the decision. Consider the first scenario. Let's say, listen to this very carefully here. Let's say over here the train accuracy is 100%. The test accuracy is 98%. Is that a good model or not? What did I say? The train accuracy minus test accuracy reference must be less than 5%. It is test accuracy is it at least greater than 80%. Yes. Then we say it's a good model. So I say yes it's a good model. Consider second scenario. Let's say over here you have 98% accuracy and over here you have 97% accuracy. Is it a good model? This must be minimum 80% and the difference must be less than 5%. So tell me is it a good model? So let me actually talk about it as a good model. Let me take the third decision. Let's say you are having a 92% accuracy over here and you have 87% accuracy over here. Is it a good model? The test accuracy must be minimum 80%. That is there and difference must be less than equal to 5%. So 92 and 87 this is less than equal to 5%. Yes. So yes yes yes we call it as a good model. Consider the four scenario. Imagine here you have 92 and here you have 85. Is it a good model? Because even though the condition one is satisfied but the condition two which is the difference less than 5% is not satisfied. In fact, you remember we call this scenario as what is called technically as overfitting. I'll talk about it. Consider one more scenario. I'm going to take one more example of this. Consider one more example. Let's say this is 90 and this is 80. This is again not a good model. In fact, technically we call this scenario as overfitting. What is overfitting? Overfitting nothing but what you call as this. This is a perfect scenario of what we call as overfitting. This is the perfect scenario of overfitting. Why? Because he mugged up the data. He did not bother to understand the meaning of the data. And because of which he was able to confidently he was able to confidently answer wrongly. That's very important point. confidently answer wrongly. That's an important thing. Confidently answer wrongly. Now, basically when a model tries to understand the relationship by mugging up the data and what you see is it's doing very nicely on what it studied but when you change the question it is not able to do nicely then we say the scenario is overfitting. Like for example, let's think about it from the human perspective. If you had a question like what are the features of Python and the respective answer for this is the question that you study. What are the features of Python? And you have the answer to that. But in the unseen part you have that question explain the features of Python and I ask you to answer it what will you say will you come to me and say Dan this is an out of syllabus question this is something which I never saw or your brain will be able to understand array it's quite logical it is nothing but the same question asked in a different way Will your brain be able to figure out that that this is more or less the same question? If you are able to do that then you are not the chhatur of three idiots. But Chhatur was not able to do that. By changing one or two words he actually mugged up the data. So what happened is if you ask Chhatur to reiterate whatever he has learned in the speech he did that very nicely. So if Rancho would have not changed his speech then Chhatur's output would have been awesome. Essentially the actual and predicted would have been the same and on the unseen also he would have been good but because he mugged up the data he did not understand when the data is changed and he did not understand the generalization and that is what is called as an overfitting. So when I say overfitting always remember the chhatur of three idiots or you can remember it this way if he's do very good on training very good but not good on testing so you can also consider that imagine you know there is a student I'm training for I'm training a student in private on his maths subject he's a 12th standard student I'm training him in private So I before his board exams I took many prelim exams for the student and I was happy to say that he was getting 96% 97% 98% in many exams which I took 99% 95%. So overall I could see that he was always getting about 93%. And so I was very happy that he is ready for his board exam. And when the board exam actually he ended up giving he could only get 83%. That's where I realized that this guy was mugging up the data. So he's very good on what I teach him but little change little change and he was not able to generalize well and this is exactly what is called as overfitting. This is exactly what is called as overfitting. So if a model is doing good on training but not able to perform equally good on testing difference of 5% is acceptable but if it's not able to do equally good on testing then we say that it is overfitting just like this board exam student as well. But let's consider I have a student Rohan. So what I tell to Rohan that hey Rohan tomorrow is your Python exam. Study and come this 80 questions. Now Rohan is a Badmas. Rohan is a Badmas. I asked him to study 80 questions. He did not study. He should have invested at least 6 hours of time for studying. But Rohan is als he he hardly studied it for 30 minutes. So do you think in 30 minutes he will be able to understand this which ideally requires 6 hours? If he is a prodigy child then it's a different story. But a normal person will he be able to understand? No. Right? He won't be able to understand. He didn't invest so much time and therefore now what will happen if I give him the test paper? Will he be able to perform good on the unseen data? Tell me one thing. Will Rohan be able to perform good on the seen data? No. That's called as underfitting. That's called as underfitting. So when Rohan has not studied only, how do you expect him to perform good? So imagine this is 7p and this is 68. We call this scenario as underfitting. Why? Because this is not an industry accepted model. Industry accepted model says that test accuracy must be at least 80%. It's not there. Difference must be less than 5%. This is there but both are not satisfied. So it's not a good model. So it can be either overfitting or underfitting. What is this? This is underfitting. Why? Because he didn't get a good accuracy on training, neither on testing. So the model must have that good at least. So this is how you end up taking a decision. So we don't want to create a model which will overfit or which will underf. So in this way there can be multiple scenarios. We can discuss it later. For now this much is more than sufficient. So overfitting versus underfitting. Now now let's understand you only have to give me your inputs. Consider I'll take an example. I want to buy a house. The price of a house is going to depend on what all factors. So let's say I am specifically looking for a 2BHK in Marali area in Bangalore where we know that the price of the house ranges between 1 CR to let's say 1.3 CR. Let's assume. Let's assume. So the price of the house depends on what all factors. So number one factor I can say that it depends on how much is the area of the house in square ft. It depends on which floor are you buying the house because the more higher you buy you got to pay for the floor size. It also depends on who is the builder who is building the property. It al so dep it also depends on over here whether the society has a swimming pool or not. See I'm not saying that whether it has amenities or not. I'm being very specific. Swimming pool. Yes or not? whether the society has a gym or not. What is the distance to the nearest shopping mall? What is the distance to the nearest school? What is the distance to the nearest vegetable market? What is the distance to the nearest bus stop? What is the distance to the nearest metro? What is the how many number of bedrooms are there in the house? Then is the society having a garden or not? Is there a temple in the society or not? And many such other factors. In the same way I never I never said amenities. I said swimming pool. I said gym. I never said amenities and concluded be as much concrete as much atomic as possible. Help me out. I have to buy a mobile phone. So the price of the mobile phone depends on what all factors brand whether you're buying Apple, Samsung, Oppo, Vivo, Xiaomi etc. It depends on the RAM. It depends on the camera megapixel. Yes, it depends on Okay. Color also. Yeah, color also matters. Nothing wrong in that. It also depends on Yes. Whether it is 5G or 4G, it also depends on uh ah OS means brand it will get covered now. Okay, I can write that but that makes sense. It depends on processor as you all said internal memory. Shwa I have the right to scold you now. The price of the phone depends on the price of the phone. Huh? Are you trying to say the price of a phone depends on the price of the phone? I know. I know. I caught you. Subconsciously you said that. So now next is yes display type. Definitely display type and and and and and uh screen size yes that's again important factor screen size yes waterproof dust proof that also I agree etc etc let's go on to the next one the third one you know there is a reason Why am I talking about this? Yes speaker mic quality that's also correct sha let's say in one hand I have apple this all is machine learning in one hand I have apple and in another hand I have orange how do you identify that this is an apple and this is an orange color. Okay, that's a good one. Noted. And shape, taste. Okay, don't say features this time. Uh, taste. Smell. Yes, that's very important. Smell. We can also look at texture. Now, if you are a blind person, you can actually feel it. How is it to touch? Feel type. Okay. And if I get into little technical stuff then you can look into the mineral content, vitamin content, vitamin C content etc etc etc things hard soft yes essence that's also important. Okay one more example then I'll conclude this topic and we'll go on a break. There is a reason why I'm discussing this. It's not for time pass. There is a reason for that because we have to start with types of machine learning once we come back and that's exactly kind of I'm creating a background for that topic. I want to talk about the very standard and the famous data set of machine learning which is called as Iris data set. So if you look over here now if you see carefully do you see all these three flowers look more or less the same and it is difficult to differentiate that these are three different flowers right? Yes. So these three belong to the same species iris but they are three different flowers and virginica. So what they had done is they had they had the data set the original data set consists of total 150 rows. What they did is for every flower they they had taken 50 sets of flowers 50 versol flowers and 50 virginica. Here for demonstration they have shown only two but there are total 50. So they have taken total of 150 flowers. Total of 150 flowers. And what did they what did they do is they measured the length of the petal width of the petal. For each of the 150 flowers they measured the length of the sele width of the sele. As you can see sele width petal length petal width and what is that flower in this way you have total 150 values. So here in this I can actually say iris data set which is sets or color or vica that depends on sele length sele width petal length and petal width. So here what I want to tell you is whatever you are trying to predict in machine learning you call it as dependent variable. Whatever variables you use for predicting it are called as independent variables. Technically independent variables are very popularly indicated by X and dependent variables are indicated by Y in machine learning. Though it's not mandatory but you will see almost 100% of the people using the same conventions. You are also requested to use the same. It's not mandatory but you are requested to use the same. So here if you see I can say that if I'm considering the price of the house as the dependent variable then this brand can be considered as first independent variable ram as second this as third this has four this as fifth this has sixth this has seventh this has eighth this has nth this as strength. Now coming up this so I asked you to study 80 questions. These questions are now called as independent variable and answer is called as dependent. So this part which I asked you to study is called as the train part. This part which I kept hidden from you is called as test part. So because of this this is X train X test sorry Y train this is X test this is Y test when you give to this model what are you giving you are giving this model to predict on X train the model predicts Y prediction on train. So you will see I'll be using the same conventions in the code. So we ask the model to study the relationship between X-ray and Y train and the model creates a I mean uh we it will study the relationship and then when you ask it questions on X train the output is what we save into Y prediction train and you compare this. Similarly to the model when I ask purely unseen data X test the model is going to predict that and I'll be saving it into the name Y predicted test. Comparing these two I calculate the test accuracy and then these two are compared to figure out whether the model is doing good or not. So if I come back to this example, you can consider over here this is independent variable. This is the dependent variable. So overall we call this as x this as y. Specifically I can say x1 x2 x3 x4. So if I have to uh uh explain this problem then I can say we have the same thing now. So if you remember we had total 150 row row number one to row number 150. So row number one to 120 which is nothing but 80%. And this is the 20%. That's the test. That's the train separate petal and petal width versus this model understands the relationship between this 120 rows which is x1 x2 x3 x4 versus y. Just like here I said the model understands okay if these are the things then it is stosa if these are the things then it is stosa if these are the things then it is versol if these are the things then it is versol and so on it understand this and ultimately on the basis of this it creates what is called as a model is created and then to this model as we discussed when you ask it to predict on x train. It predicts this. When you ask it to predict on X test, it predicts this. And then as we decided using this, you get the rain accuracy. Comparing this, you get the test accuracy. Technically, this is X train, Y train, X test, Y test, Y prediction train, Y prediction test. And that's how you actually end up creating a machine learning model quickly. So if I have to just take it here, that's how we have it. I'll do one thing. The case study will be Iris data set modeling. Let's do the necessary imports. I import the numpy pandas macot liv se. I also import warnings. Then I say getting the iris data set. This is the data set. Then I say I divided the data into x and y. So X is expected to be all of this. That's what I said from the data frame. Everything except the species column goes into X. And similarly Y is only the species column. So this is Y. Then I say doing the division. So you can see over here we add this data. We divide it into X and Y. Then further into this because I said you can see X train shape 120A 4 Y train shape 120 comma one row one column only X test shape which is this 30A 4 and Y test shape 30 comma because I divided it into 80/20 20% in the test part so 8020 okay I'm creating a machine learning model so I'm creating a model co of logistic regression. I'm asking the model to fit. Fit. Fit means study. Understand the relationship between Xray and Y train. That's exactly what I used to say. Understand the relationship between Xray and Y train. So I'm asking you to do that. And it says I'm done understanding. Then I say okay then let us predict on the training and testing data. So I say my y prediction on the train is model predicting on x train similarly model. So I got the model to predict on what it studied into this what it has never seen but similar to what it studied into this. Then I would like to calculate the accuracy. I'm calculating the accuracy of the training and I'm calculating the accuracy of the testing. You see train accuracy and test accuracy and then I'm printing it 97% accuracy on the train. 96% accuracy on the test and this is the end of this notebook. Essentially in any machine learning model these will be the steps. I will once again go through it. First we did the necessary imports. We had been doing it earlier as well. That's not new. The data set I loaded it's already available in seaborn. So I used that and the data is loaded. Okay. Then next thing is we have to divide the data. This is my independent variable. This is my dependent variable. >> Yes. >> So ideally how many rows are there in the data set? I can actually say DF do.shape 150A 5. 150 rows and five columns. So this 150 comma 5 is taken and then you divide it. So obviously I said divide it in such a way that it has to be 20% data in the testing. So 150 rows are there. That means 120 will be over here. 30 will be over here. So that's what I did over here and I get four parts X train X test and Y train Y test. Then then I say I'll create a logistic regression model. So I imported the logistic regression model and I said logistic regression please go and understand the relationship between X train Y train. So that's exactly what is happening. Once it has understood I say logistic regression go and predict on x train predict on this part and I get the prediction here we call it as y prediction train this one logistic regression go and predict on x test which is this part saved over here I call it as y predicted test then I say calculate the accuracy by comparing calculate the accuracy by comparing by train and y predicted train. So y train is this part, y predicted train is this part. Compare this and calculate accuracy. Similarly, I said calculate the accuracy by comparing y test and y predicted test. This and this. So I get the train accuracy. I get the test accuracy. That's what I printed. Train accuracy printed test accuracy printed 97.5% on train and 96% on test. Done creation of the machine learning model. Let me come to you and ask you overall did you understand what I did? overall only I don't want in-depth understanding motor whatever I taught you the same thing is required in the code right this is one of the very very important thing very very important thing right so you understand that whatever theory I am teaching you is not just a theory but it's actually code only you'll not understand code this is what we have and that's where I close this and we will go on a break. So just once again to confirm, I hope you are convinced with the overall flow. Okay. Now, now friends, we understand what is machine learning. We have also seen a very raw implementation on Iris data set. Now let's talk about machine learning. So friends, machine learning is broadly divided into two types. First is supervised learning, second is unsupervised learning and the third is reinforcement learn. Now supervised learning is a learning where a supervisor is available like right now if you are learning something from me I can call it as a supervised learning why because I'm the supervisor I am available so if you have any question ask me here only I will clarify it that's called as a supervised learning a supervisor is there. What is unsupervised? Where a supervisor is not there. Example, you are watching some video on YouTube. So let's say if you go to YouTube and here you are saying that I want to do machine learning on data set. And let's say you're randomly picking up this info this data. So the author over here let's say by the channel hackers realm he says he has analyzed the Iris data set with various this this this this and he started explaining you watch it you are over here he read the data did some analysis then after that this is the correlation matrix then he is getting into dividing the data extend extension all these things done and you have this. Now let's say at some point of time you are having a question but is there anyone to help you out on this? You have a question why is this warning coming? You wanted to ask in my live session you can ask me that why are you getting this warning? How to get rid of this? But on a YouTube video, what can you maximum do? Maybe you put in some comment below and wait for this person to answer and you wait for this person to answer. So it is an unsupervised learning. You don't get the live support. So technically we say supervised learning is a type of learning where you have labeled data. Labeled means independent and dependent variable both are available. And unsupervised learning is a type of learning where you have unlabelled data which means only and only x is given and you need to find or derive y. You need to find or you need to derive y. That is exactly what we call as unsupervised learning. What is supervised learning? Where you have access to the supervised data. So how do you actually do this? Supervised learning is further divided into two parts. The first one is a regression problem. The next one is a classification problem. So regression is the one where the dependent variable is continuous in nature and classification is the one where the dependent variable is categorical in nature. Regression is the one where it is continuous and classification is the one where it is categorical in nature. So over here example if you come back here see example one and two what you are predicting is the price of the house it is continuous because the price of the house as I said can be anywhere between 1 to 1.3 cr any value it can take or price of the full also so I may buy it as 20,000 someone buys it at 20,300 someone else buys it at 30,145. So there are various possibilities over here. So now similarly uh the other two right. So example one and two is where you are predicting continuous value. It's not the same everywhere. In classification the target variable is categorical in nature. Like if you consider apple versus orange or you consider Irish data set there are limited number of categories. So we call it as a classification problem. I repeat whenever you have labelled data that means independent variable and dependent variable both are given you call it as a super problem. So if the dependent variable is continuous that is called as uh regression and if the dependent variable is categorical that is called as classification. That's one thing over here for we can also say well Amit what we used was a classification approach only logistic regression is a classification algorithm. It is not a regression algorithm. Give me some time I will come and justify on this but logistic regression is not a classification algorithm. what we use was regression. Uh what we used was classification only. It's not a regression algorithm. I know that I know that I I usually compare when we study these algorithms and all. We'll be discussing all of these things of course. Uh okay. Now this is one part of the thing and what all algorithms come inside that let us actually write it. Uh no I'll not write it. Why to unnecessary invest time? we will do it in a smart way. Now see we understood that regression algorithms are the one where you are going to have the target variable to be continuous in nature and the classifications are the one where you have target variables to be categorical in nature. Then after this the next thing over here unsupervised. Now unsuper is the one where only x is given but y is not there. Now that is little weird right? How can it be possible that dependent variable is not there. I'll tell you let's take an example to understand this thing. So what am I considering over here is let's say I am collecting we are how many in the we are 55 in the class right now right? So I am saying that everyone please come and deposit your phones on this table. Please come and deposit your phones on this table. So I'm depositing my phone. You deposit your phone. In this way many people all the 54 phones are deposited on my table. Okay. So we have total 54 phones. Now the thing is I invite Pritham on the stage and I ask Pritham Pritham these are 54 phones. I want you to group it on the basis of similarity. So Priam what can you what will you as an individual consider for grouping this phone based on some similarity factor. Okay. So let's say Pritham says brand. So if Pritham is dividing the phones on the basis of brands in our class considering there are 54 people how many groups is Priam likely to get likely mot what do you think? How many groups will priam get? >> 3 to 4, 2 to 3, 10, 4 to 5, 5 to 6, 6 to 7, 5 to 10. Yes. Let me take a realistic figure. Let's say he gets around five to seven groups. Samsung, Apple, Oppo, Xiaomi, Vivo, nothing. 5 to 7. Okay. I go to Rahul. Rahul says no no no D sir I will not group by brand. If I have to group it you know I am going to group it by color of the four major color. So it's not that white and off-white I'm going to separately white, red, blue, yellow, pink. So if he does like that maybe he gets around again five to six groups someone someone else comes he say that I am going to group it on the number of cameras. Number of cameras means see in in my phone if you see there are two cameras over here. So I'm talking about this as the number of cameras. So if I'm grouping it on the number of camera not the camera megapixel how many groups will I get in my group in my class maybe three to four not more than that someone says no no I am going to group it on the basis of internal memory is the memory half GB 512 GB 1 TB etc etc So he gets again around three to four groups. Now the thing is who is wrong in this case? Whose grouping is wrong? By brand, by color, by cameras or by ROM? Whose grouping is wrong? No one, right? Yes, no one. That's exactly what is the thing. So consider over here. Now you know how is the data that you are having. So I have collected the data of 54 phones right. So phone number one, phone number two right up to 54 and there can be various features you know the first feature could be what is the brand of the phone. So let's say if I'm talking about me let's say it's Apple then the second feature could be what is the model of the phone. So let's say I'm saying uh iPhone 15. Then the third feature could be what is the color of the phone. So let's say the color of the phone is green. Then fourth feature could be what is the RAM of the phone which honestly speaking I don't know what is the ROM of the phone. In this way there are let's say around independent variable 30 independent variable talking about various features and that is it what we have. So let us say we decide to group it as per the brand then uh okay brand okay let's say brand or let's say decide to group it by any other thing. So what will happen is anywhere you know let's say if you're grouping it by brand then anywhere you see uh the phones which are uh by Apple. So let's say I say that wherever I see Apple phone I'm going to call it as group one. Wherever I see Samsung phone group two Xiaomi phone group three. Oppo phone group four. Nothing phone group five view phone group six. So here my dependent variable is what? Who is finding I am finding dependent variable. Earlier it was given here I will be assigning group one. Maybe this is a Samsung phone I assign group two and so on and so forth. I'll assign any one of these values over here. Once I have this then I can convert it into a supervised problem. But in unsupervised you are not given any kind of in unsupervised we are not given any kind of this that's important. Are you all understanding what is unsupervised learning? You don't have a target variable. You don't have a dependent variable. That is what is important. So now in unsupervised learning there are uh following domains. The first domain is called as clustering. And here you have the algorithm gains. You have hierarchical clustering which we just saw you have DB scan algorithm. The next one that you have it is called feature selection algorithms. Here you study algorithms like recursive feature elimination and variance inflation factor. Variance inflation factor you'll study that it's better we talk about it later. Third is you also have dimensionality reduction algorithms. So here you have PCA LDA. So you know when they are used so consider scenarios where the number of independent variables are not just 100 200 but you have the number of independent variables as 1,000 3,000 5,000 these many are the number of independent variables that you have and in that case it becomes a challenge. So how do you actually reduce it? How do you actually reduce it? And that's where that's where the challenge starts. So these algorithms are mainly used for reducing the dimension. Now there is a difference between feature selection and dimensity reduction. It's little difficult to make you understand right now but we'll be understanding it as and when we go ahead. Then the fourth one is what we call as analy detection algorithms and the fifth one. So there are five types inside this. Here there were only two types but here there are five types association rule mining algorithm where you have algorithms like a prior or FP That is how you study this. Then the next type of learning is what you call as reinforcement learning. So what is reinforcement learning? Well, reinforcement learning is learning from our own past experiences. So if I take example let's say about the Tesla's autonomous car. Elon Musk and his team has already trained the model with some pre-built knowledge and using that knowledge it takes a decision but the best part of that is taking decision was still fine but after taking the decision after taking the decision it will also automatically learn in real time setting. Like for example, let's say I was driving a car some point of time on a highway. This was the going way. This was the coming way. And this was my car. There was a truck which was here. I gave a right indicator and I moved my car over here. But this truck person should have break. He instead accelerated. And when my car was here, he dumped my car here. So that's a learning that there are some crazy idiot nonsense drivers who don't understand that they have to break if there is a safe distance and allow the person to go. So these all are nothing but what we call as learnings. The car also learns from experiences. So whatever knowledge it has, it keeps on updating its knowledge base and that is what is called as a reinforcement learning. Learning from your own past observations. This is another 40 hour of syllabus. In our syllabus or in anywhere, right? Reinforcement learning is not a part of our syllabus. Our syllabus is mainly talking about supervised and unsupervised part. Now coming on to the supervised algorithms as well. If you talk about this see that's why I didn't write it down. If you see following are the algorithms. Linear regression, K nearest neighbor regression, decision tree regression, support vector regression, add boost regressor, gradient boosting regressor, random for regressor, extra regressor, AN regressor and here amoge logistic regression it's a classification algorithm LDA K nearest neighbor classif classifier, decision tree classifier, support vector classifier, add a boost. Yes, that's correct. You heard it right. They are available in both the cases. Adaboost regressor, Adaboost classifier, gradient boosting regressor, gradient boosting classifier, random forest regressor, random forest classifier, extra tree regressor, extra classifier. Similarly for ANN and there are many more algorithms but these are the popular ones which they have listed over here. So I come and quickly put this and that gives you some idea about the available algorithms. Just read it once and now also read this chart overall. And yes, so now we are good to get started with our very first algorithm which is called as the linear regression algorithm. Enough of introduction. Let's get into the very first algorithm. So I come down and this is where I give you the introduction. M honestly speaking no but you can create a model in Java or C++ as well but created in Python can be used in uh Java or C no that won't be compatible right and why do you have to create it in Python or Java for inferencing see if it has understood everything using the language as Python uh of course it cannot be programmed with any other language. Ah that's too early actually. Uh we are going to get into that only. In fact now when I said we'll be starting with the machine learning algorithm we actually get into that. That's correct. So so friends uh let's get started with the very first algorithm which is called as linear regression. So, so, so here we are. Linear regression. It is a supervised machine learning algorithm. This algorithm can be used if and only if there is a linear relationship between independent and dependent variable. So what is exactly meant by linear right? So I said this algorithm can be used if and only if there is a linear relationship between the two. So what is meant by linear? See here consider this. I'll consider several scenarios. This is independent variable. This is dependent variable. Let's consider the temperature versus sales. Now overall we understand that as the temperature increases let's consider the sales of ice cream as I always consider increases. So this relationship can be more or less modeled on a straight line something like this. So I say that overall we see that the relationship is linear. That means I can say that as x increases y increases or I can say x is directly proportional to y. But as one increases if other increases we call it as linear. You have an independent variable you have a dependent variable. Consider one more example. Let the x-axis be the number of cigarettes you smoke and yaxis be the number of years left in your life. Do you all agree that the more cigarettes you smoke, the more early you die? Now don't give me any odd examples that he has survived so long he is smoking. What is the scenario in most of the people? Do you all agree that the more number of cigarettes you smoke, the more early you are likely to die? Yes. So the thing is is this relationship linear? Because here we see that as x increases y decreases. So I can also say that x is inversely proportional to y. But I can still say that this is linear because it can be represented on a straight line. This also can be represented on a straight line. Now I don't know what can be the example of this but just tell me do you see this as a linear relation this one? No. So this is something which is not linear. Essentially if and only if the relationship is like this then only it can be used. Else it doesn't make sense in going for this. Else it doesn't make sense in going for this. That is something which is very very important. And that's exactly what we call as linear also nonlinear. So if your relationship is linear then linear regression can be used else you can't use it. As simple as that correlation is different from the linear regression. And the reason I say correlation is different from linear regression is because correlation can be between any two variables. So I can find out the correlation. That's a very good point. Correlation can be between anything. It can be between two independent variables X1 and X2. It can be between some other independent variables. But a regression will always be between independent and dependent variable. Although mathematically you are going to find a single value only but correlation is just going to tell you whether they move together. regression will be helping you to predict the unseen value. So that is how they are different right? Yes. So we we are going to talk about that you will actually understand what is that. Got it. Very good. So now now now uh yeah we'll at least let's complete with the algorithm for uh understanding part of the algorithm. Remaining things we can at least take it later. So let us get started with a very basic understanding of regression algorithm. So some basics going back to school we studied in school something which is called as the slope of the line. Some of you might remember it some of you might not remember it. I'm not assuming anything whether you know it or not. So I'll get started from the scratch only. So here in school we studied that if this is a line then every line has a equation which is of the form y = mx + c. y = mx + c. This is what we had studied here. M is nothing but the slope of the line which is given as change in y upon change in x which can also be written as y2 - y1 upon x2 - x1. So what is y2 and what is y1 on this line? If you assume anywhere any two points on a line on a line not on a curve on a line anywhere you assume any two points let's assume these two because these are directly available what are these 86 23 so I can assume this to be x1 y1 x2 y2 Which means that if I am doing y2 - y1 it is going to be essentially 6 - 3 upon 8 - 2. So m is going to be is equal to which is 0.5 and if some of you say that no no d why are you not assuming this as x1 y1 and this as x2 y2 so if I assume it like that then what is the final answer? Tell me quickly what will be the final answer. If I assume this as x2 y2 this as x1 y1 h same because this is -3 upon -6 which is still 0.5 in fact even if you assume any other two points also it will always be 0.5 and the next is what is this c is nothing but called as y intercept the distance from zero to wherever this line intersects the y-axis that's called as c. So if this line would have been intersecting here then c would have been this distance on a negative side. So it's called as a y intercept. So in my case the equation of the line becomes that is what is the equation of the line? uh what what do you exactly mean by the slope of the line? That's another thing which I want to say. So slope of the line means for every one unit you travel on x-axis. How much do you climb on the y-axis? Means see here from here how much I traveled on x-axis three uh six six units right I moved from 2 to 8 total 6 units and how much did I climb on the y-axis if I go like this and like this then you can see I traveled three because this is going to be three and this is going to be six 6 - 3 is three so that means I can say that if I am traveling three units. That means if I'm traveling 6 units on x-axis on the y-axis I am climbing three. If I travel one unit on x-axis I should ideally travel half of it. Check it out. I'm here from two. If I move to three, how much do I climb up? 3 to 3.5. Half of it. That means if I travel 12 over here, I travel six. If I travel uh 16 over here, I travel 8 and so on. That is exactly what is called as M. For every 1 minute you travel on x-axis, you travel half on the y-axis. That's essentially what m is. Therefore, the m came out to be 0.5. And this is what is the basic equation of the line which we have understood in our school days. Yes, this is our basic understanding that we are having from our school days. So now I'm going to put that over here. Now uh mathematics of linear regression I'm going to explain you next time once again. Uh this time I'm just going to give you a quick heads up. So now consider over here. Okay, let me ask you know uh before we start with the mathematics uh let me ask you if uh what is the slope of the line? Minimum and maximum value of slope of the line. What is the minimum and the maximum value of slope of a line? Okay, let's see. See here you see this is the dynamic figure. You see the slope of the line is getting calculated here. If I make it more up, it crossed one. Okay. See here, what did I say? I took the data set of 150 rows, divided it and then I divided the data into 80% and 20%. Which was done here. Here I did say the test data must be 20%. So by default this becomes 80%. I want some of you to try out 8515 ratio. I want you to try out the 7030 ratio. I want you to try out the 7525 ratio. I want you to try out 6040 ratio and every time you'll be getting the accuracy. I want you all to make a note of all the accuracies and tell me which ratio is giving you the highest accuracy. So you can try out any you can try out any value of test size from 0.35 and below every time decrementing by 0.05 0. So 35 30 2510 I want you all to try out these all values every time making a note of this accuracy and tell me which combination is giving you the least accuracy. Meanwhile, I'll just get this exported. Let me talk about what is exactly linear regression and when should you use it. So let me give you an heads up around what is linear regression algorithm. So by saying linear regression it signifies relationship between a dependent variable and one or more independent variables. Example price of car is dependent on the length of the car. more the length of the car, higher is the price of the car. This is called as a linear relationship because as the value of independent variable increases, the value of dependent variable also increases linearly. Price of the car is dependent on the horsepower of the car. As the horsepower of the car increases, the price of the car also increases. This is also called as linear relationship because the value of as the value of independent variable increases the value of dependent variable also increases. Price of the car is dependent on the weight of the car. The explanation goes in the similar way. As the value of independent variable increases, the value of dependent variable also increases. Price of the car is dependent on the mileage of the car. This is called as linear relationship because as the value of the mileage of the car increases, the price of the car decreases. But here the relationship is negative in nature unlike all of the previous examples where it was positive in nature. So basically we can say so one thing is for sure you might have seen every time I have been writing one word dependent variable independent variable this one word you might have seen I have been using. I never talk about two independent variables. I always talk about one independent and one dependent. So here independent dependent that's very important independent dependent independent is dependent. So if I'm considering temperature versus sales of ice cream, we know that as the temperature increases the sales of the ice cream also increase. So roughly we can see that this relationship can be plotted across a straight line because you can see the trend. So that means as temperature increases sales of ice cream increases. That means I can say that X and Y is directly proportional. Yeah. So I was saying uh if you consider the same example like we discussed earlier this is the number of cigarettes a person smokes versus the number of years left in his or her life. So we know that the more number of cigarettes you smoke the lesser are the number of the years which are left in his or her life. So here also we say the relationship is linear. Here also we say the relationship is linear. Here there is a positive relationship and here there is a negative relationship. Negative means as x increases y decreases or x is inversely proportional to y but both are linear. But if you have a data which is like this. Now in this scenario this is the kind of relationship that you have between independent and dependent variable. This is what you call as nonlinear. Now the very important thing is I want to talk about the difference between regression and correlation because regression and correlation looks very similar. Correlation it defines the association between any two random variables. regression. It defines the association between dependent and independent variable only. Example based on DNA and blood group match a child will be associated to a parent or not. So in correlation you just try to find out the value which will signify whether these two things are having a tendency to go together or not. But in regression you are using it as a machine learning problem to predict the future values. So in correlation it describes a linear relationship but in regression you actually look at the best line on the data. So there is a difference. So basically correlation can be between anything. So you can just consider in simple words that correlation is between independent variable versus independent variable. Regression is between independent variable and a dependent variable because they are very very simple and you'll understand it in more detail as in when we get into the implementation of it. But for now you can remember the most important difference as correlation can be between any two independent variables and regression will be compulsorily between independent and dependent variable. ever seen. So we have already seen how correlation uh is different from the causation but uh you know let me just try to give you some more uh differences in between that. So, but we'll but you'll understand it 100% as in and when you go ahead let's talk about it. Correlations versus regression. Correlation measures the strength and direction of linear relationship between two variables. Regression predicts one variable based on the other. We call this purpose type of relationship symmetric in case of correlation. Correlation between is same as between in case of regression. Regression predicts y from x. So the direction matters. The direction matters. So basically correlation between x1 x2 is same as between x2 x1. But over here in regression the relationship between x and y is not as same as between y and x. I'll give you a clear example as well as we go ahead. The third point >> output in correlation the output is a single value between min -1 and + 1. In regression, we get a equation of the line y = mx + c. Slope intercept causality. Correlation does not imply causation. In regression, it is often used to infer or test position variables role in correlation. There is no distinction between independent and dependent variable in regression. There is a clear distinction where X is the independent variable and Y is the dependent variable. Units correlation is unitless standardized. In regression the units matter. output is in units of y. Let's try to clearly understand it with the help of example. Example one, height and weight of adults. Correlation will measure how strongly height and weight move together. And let's say they give a value of 0.85 indicating a strong positive correlation. This means that taller people tend to be heavier. Regression will predict the weight on given height using regression line. Here you get an equation after you apply the machine learning algorithm which looks like y = mx + c. In this we have to predict weight. So weight can be 0.9 multiplied by height + 40. A person of height 170 cm is likely to weigh. This will be 170 which is finally equal to 193. Is he approximate? So you can see we are actually finding out something using something in recorrelation. We are just comparing. If I take one more example of Study time versus exam score. In correlation, we check whether more study time leads to better scores. Let's say we got the value of correlation as 0.7 which shows moderate positive. This means that more study hours are generally associated with better results. In regression, we predict exam score based on the study time. So again, you'll be getting an equation which will look like this one. y= mx + c where y is nothing but the exam score let's say which is pi into study hours plus 50. So a student who studies for 4 hours is predicted to score exam score is equal to4 + 15 which is equal to 70. So you can see over here the use case of each of this is completely different. You saw this example of height and weight study time versus exams score the uses are completely different. During regression we are trying to predict in correlation we are trying to see how they are going together. So therefore I say coming in 10 seconds. Okay. Early morning uh when you get up now it's all jam. In an hour it becomes normal. Correlation tells us how two variables move together. Strength and direction. Regression tells us how much one variable changes when the other changes. Hence used for prediction. Yeah, that's how it is for now at least. That is how it is. Now let's try to understand directly getting into the mathematics. So here paining a lot. So I was saying some basics of what we have done in the school. In school days, we understood that the equation of the line is written as y = mx + c where m is nothing but the slope of the line which is nothing but change in y upon change in x which is y2 - y1 upon x2 - x1. Yes, no need to be smart by giving suggestions over here. This is very basic. So please focus only on understanding anyway that's school level but I remember when I started machine learning I had forgotten this also so this is specifically for the students who are as weak as I was right those who don't remember most of you might be remembering but trust me I had actually forgotten how to calculate the slope of the line so now how do you calculate sorry that is the slope of the line And C is nothing but what you call as Y intercept. Now how do you calculate M? M is calculated as change in Y upon change in X where consider any line on the XYaxis. You assume anywhere any two points on the line. Yes, that's important. anywhere any two points on the line. So let's consider the points which are already here 86 23. So I can assume this as x1 y1 this are x2 y2 so y2 - y1 is 6 - 3 upon x2 - x1 is 8 - 2 which is 3 upon 6 which is 0.3 Similarly you can also say that no no I'll assume this has x2 y2 this as x1 y1 if that is the case then it is 3 - 6 upon 2 - 8 - 3 upon - 6 which is again 0.5 and even if you assume this point over here maybe if you consider this as 12A 8 then still that will be 8 minus let's assume this as the point so 3 upon 12 - 2 which is 5 upon 10 which is 0.5. So essentially you assume this points anywhere the answer will be this and wherever this line intersects the y-axis that distance is called as C which is nothing but y intercept. So imagine this line would have been like this then C would be minus2 in that case but our line is something like this. I would like to ask you one question. uh the slope of the line ranges between what and what? Slope of the line ranges in between minus infinity to infinity. So m can go on from minus infinity to plus infinity. Okay. So these all was just a quick context setting once again. And now I get on to the linear regression. Now let's try to understand how linear regression actually works. So here let's assume that on the x-axis and y-axis we have the following data. So let's say on the x-axis I have the area of the house in square ft and on the yaxis I have the price of the house. Do you all agree as the area of the house increases the price of the house also increase in a particular locality? Yeah. in a particular locality. Right? So let's say the area of the house is 2,000 square ft². Let's say I'm assuming the price to be 2 k. If the area of the house is 1,500 square ft², I'm assuming it to be 1.5 cr and so on till the end. So let's assume the same thing over here on the x-axis. I am assuming the area of the house and on the y-axis I'm assuming price of the house. So obviously area cannot be 246. So let's say into 1,000 square ft² and here let's assume this is 2CR 4C etc. So in this case that's my independent variable and that's my dependent variable. What will I do is I will end up plotting all these values on a scatter plot. And it is expected that as one increases the other also is expected to increase. Now what is the end goal? See the end goal over here is now that we have this data we want to draw a line. The machine learning algorithm tries to draw a line in such a way that the line should pass through all the points. Is it possible to draw a line which passes through all the points? Yeah, I think this is how we can try it. But is this a line? No, we just understood that a line is like this, like this, like this. Anything which is straight is only a line. This looks like some stock market graph. So they say let's try to draw a line which passes through maximum points and the ones which it is not able to pass through is as near to the points as possible. It should pass on through maximum points and other points should be as near as possible. So is this line passing through maximum points and others are as near to it as possible? The first line? No. Is the second line better than the first line? Better. Okay. Is the third line even better than the second line? Yes. Is the fourth line even better than the third one? If I assume the fifth line, can you assume this is the best so far which is already drawn. So among so many things if you look at this fifth one can I say this is the line of best fit right? Yes. And obviously if it is a line we know that it will be having an equation and since we already worked on it we know that this line has a equation which we just populated before coming to this where it was having the M value calculated as 0.5 and what was the C value calculated as two. Now the thing is how does this equation help? Let's say you created a linear regression model which is talking about predicting in this way. A friend of you comes to you and says that dash you are a data scientist. I want to buy a house in this particular locality and I am looking to buy a house which is 3,500 ft² which is 3,500 ft². Can you let me know what is the price at which I should buy it? And I say yeah definitely. What will I say? I have this equation 0.5 into 3.5 + 2. Why 3.5? Because it's into,000. Can you tell me what does this evaluate to? Half of 3 is 1.5, 1.75, 3.75. Correct. So, can I say he will buy it at the best price to buy is 3.75 CR? Yes. He said he wants to buy a 3.5 CR house. So this is two, this is three, this is 3.5. And I say that you want to buy a 3.5 cr house. Then this will be the price of best. And you can see this is two and this is four, this is three, this would be 3.5, this is 3.75. That's perfectly this. So this is where I did show it to you geometrically. This is where I did show it to you algebraically. Are you able to understand both the approaches? Some important points I want to discuss now. Some very important points I want to discuss now. But before that, let me just capture this screenshot. The equation of the line. As we just understood, a friend of mine comes to me and says that Dan, I am looking for a 17,000 square ft² house. Could you help me? What will be the price of the house? So, what will be the price? I want to hear it from you because we have just learned how to do that. So, what will be the price of the house? Okay, 10.5 CR and this is exactly where I want to put forward my point in this scenario. I completely understand that the equation works but I want to raise a warning over here that the right answer to this should be I don't know now you might be thinking why because here if you see we are extra extrapolating and extrapolating is risky. Now you'll be like what is extrapolating? See if you remember the data points which we had from the earlier figure. See I had drawn it like this. This was how the data was. There was not even a single house in our data set which we did draw over here when plotted. There was not even a single house in our data set which was above 14 cr and here they are asking us to predict for 17 cr. Am I correct? Did this pink dots go anywhere above 14 cr in the previous figure? No. Right. And here we are asking it to predict for 17 and I say it is risky. Now you may say that Dan sir but looking at this trend as the price of the house is increasing sorry as the area of the house is increasing the price of the house will increase. So consider the area where you are living so lowerass middle class or maybe higher middle class. I'm sure not ultra rich but what I want to say is imagine the area in which you are living. There will be lot of 1BHK apartments, lot of 2BHK apartments, lot of 3BHK apartments. These are the common needs of the people. If you build up a 7BK apartment, let's say I am a builder who comes to your locality and I build up a 7Bsk apartment where most of your paying capacity is somewhere between 50 lakhs to maximum 4CR and I'm building up a 7 BHK apartment and I'm selling it at 8CR. Do you think there will be even a single customer capable of buying it or a ultra rich person comes and buy it in a locality where people like you and me stay? Do you think like that the sales will happen? No. That is why though the price that you predict may be correct but no one will buy it. So there is no business sense over here. I'll take one more example. Consider over here. Consider over here the same example of temperature versus sales of ISM. So we know that temperature is 10°, 20°, 30°, 40°, 50°. As the temperature increases, sales of the ice cream increases. No doubt about that. Let's say the temperature becomes more. Temperature becomes even more. People will even stop getting off their houses. Even the Swiggy Zomato will stop their deliveries. And the pattern would be like this. So what you thought where you know you plotted a linear regression algorithm because you had data given up to 50°C and then someone said that okay if the temperature is 60° what will be the sales of the ice cream? What will you predict? You will predict okay it's 60° and you want me to predict the sales of the ice cream. So 60 is here and this goes here. So according to you the sales of the ice cream is this which is very high. But that is wrong. That is wrong. The actual sales of the ice cream is this. This is the actual sales of the ice cream. But what he predicted was this. And this is exactly what is called as extrapolation. You assuming that the trend will continue. That's risky. It may continue. It may not continue. So the idea is very simple. You should always ask the machine what it has studied. Not out of the range what it has studied. So if a machine is being taught something, you should not ask him to predict out of the box values. Like for example, let's consider this application which I talked about. I remember this streamlit app is what I had shown it to you earlier. Am I correct? Did I show you this app? Yes. Very nice. So consider now in this data in this data let's consider you had the iris data set not iris you had the tips data set. So that data set which you had consisted of you know we were predicting the tip given based on four independent variable and one of the independent variable was total bill I remember in the restaurant when I did df.escribe described the total bills minimum value was $1 and the total bills maximum value was $17. I remember that very well. The size of the people coming the minimum was one and the maximum was six. So if you ask a model that hey can you predict if the total bill is $25 and if 12 people visited and rest all things you gave in range can you predict the tip given it will predict but the chances of it being right are not so great one should not do that tell me one way how we can avoid it as a customer the customer should know that it has to not give any value above this or below this. It has to be compulsorily between 1 to six. How can you think of handling this? Just a logical move. Let's say this is the problem that you are facing. What input will you give? The customer should not give any value out of this range or out of this range. What value can what what can you do? It's nothing related to machine learning. It's nothing related to machine learning. You cannot change anything in machine learning. Think about what change can you make to this user interface. Yes, that's nice. You can put in an input validation. That's one of the very very important thing. You can put in an input validation. So precisely if you see I have put in over here plus and minus plus and minus should be there but the maximum plus like for example the minimum is zero. I cannot you know you can see the minus is frozen right now the minus is frozen. I cannot go below that but the plus is available. See minus is not available. Plus is available and it is incrementing. This plus should have the maximum range up to uh 17 and this minus should have minimum range up to one. Similarly here in size of the party see this is going up to this but there are no cases where the model got a chance to predict it and this is extrapolation that is risky. So tomorrow if I say 300 people are going, it is predicting the tip as $70. But that restaurant cannot accommodate 300 people. It has only a seating capacity of maximum 30 people. Similarly, I give the gender as M. The model was taught either male or female. If I'm giving M, it should not accept. It should not accept. It should not accept. Or if I say m, it should not accept. So these are the things that a user interface person will have to take care of. But again it is your duty to tell it because all the machine learning algorithms work on a very simple fun garbage in garbage out. All right. And extrapolation can be risky. Okay. So that was one thing. The next thing, the next thing is you have seen this is the last thing which I want to say. We just saw okay that in this example we had one independent variable versus one dependent variable. So for that we did go for a scatter plot which was two-dimensional in nature and that gave us the equation y= mx + c. Consider the scenario where I have two independent variables. So this time the scatter plot would be three-dimensional in structure and the equation of the line would be m_sub_1 x1 plus m_sub_2 x2 + c. Consider the scenario where I have three independent variables. So the scatter plot will be see one one independent variable 2D plot two independent variable 3D plot three independent variable 4D plot and do you know how does a 4D plot look like? No, as a human we cannot see anything about three dimension. But this equation can be m1 x1 plus m_sub_2 x2 plus m3 x3 + c. In the same way consider I have n independent variables. So the plot is n +1 dimensional. I cannot see how it looks like but the equation will be m1 x1 plus m_sub_2 x2 plus mn xn plus c. So essentially what I want to tell you is this figure may not be of help to us when we go anywhere about 2D because 3D plots are also difficult to visualize. But this equation is very very important and you will see this equation coming into action when we actually get into the implementation part of it. So I will now get started with this and let us get into it. So first of all the idea is to implement simple linear regression and say let's do the inputs import compile pandas ml c bar then I would say uh input warnings done. Then after this I will also be reading the data frame. So I say data frame is equal to query dot read CSV. The data is already available here. So you can see the data is very simple and straightforward. the number of hours student has studied versus the score that they have received in an exam. There are 25 students in this class. So to start with that's an awesome problem statement. Going forward we'll be looking into complicated ones only. But first one is very good for understanding. So yes. So description of all the columns you see the first column is hours with the number of hours a student has studied versus the scores that he or she has received. Now what did I say? What is important? We are here to implement linear regression and if you look into the shape of the data it has 25 rows and two columns. It has 25 rows and two columns. What do we want? I want first of all there must be linear relationship between the independent and the dependent variable. So let's do one thing. Let's plot a scatter plot to visualize the relationship between ours and this. So I say sns do.scatter plot. The data frame is df. X and Yaxis as you see and title is there, X label is there, Y label is there and you can see this is how the scatter plot is. Do you see over here there is a linear relationship between the hour studied by a student versus the score that he or she has received. Yes, that means linear regression is probably a good candidate to be tried out. In fact, instead of this, if I copy the code and instead of scatter plots, if I go for join plot, everything I keep the same, but kind I'm keeping as scattered, right? So this is the same thing. I just replace scatter plot by joint plot. But kind is scatter. I see the histogram also. But I don't want to go for this. I actually go for kind is equal to and uh see when I just said join plot of the same thing you see this kind. So I was saying you can see over here when I say kind is equal to regression we can see the line also this was just for plotting of course this is not machine learning but it is kind of finding out the relationship. Okay. Now see over here, now that we have this data set, this data set consists of 25 rows and two columns. The first thing is to separate out X and Y. So the hours must be the independent variable and score must be the dependent variable. So I will have to separate it out. Now how can I separate it out? Well, I can say, you know, I want to show you how do I select the hours column like this. That's how I select the hours column. Enter hours column is there. First thing, what is the data type? When I'm checking the data type of the hours column, it is series. And now if I put the same thing like this, what is the difference in between the two syntaxes? What is the difference in the two syntaxes? Data type and the double square bracket. That's the difference. Now here I say x is equal to df of hours and this. Now this is important. What did we do? We separate the feature or I can say independent and dependent variables. X is the independent variable. Y is the dependent variable. It must be a 2D array. And here it must be a 1D array. That is the rule of machine learning. You know why X must be a 2D array? Because here you are having one independent variable. But in real life there will be more than one independent variable in almost every scenario. Yes or no? In real life there will never be one independent variable. There will be many maybe more than 20, maybe more than 100, maybe more than 500 also. That is the reason this has to be 2D and therefore double square bracket. Remember this. We will not go for single square bracket. So X and Y separated. I can look into the shape of X and Y. X is 25A 1 which means 25 rows and one columns. And Y is 25 comma which means just one. So this is what we have. Now what is the next thing that we have to do? So for now we had this data which consisted of this X and Y. We separated X and Y. Now it is the time for us to divide X into two parts. The upper part will be called as train. The lower part would be called as stressed. So this will be called as extreme and this will be called as X test. Also we need to divide Y also into two parts where this is called as Y car train and this is called as Y test. We had discussed all these conventions yesterday. Am I correct? Did we discuss all these conventions? Yes. Now follow the two ink colors which I have used. Now if you have to do this, machine learning says you want to do that, no worries. There is a class called as train test split. In machine learning, whatever you want to search for, whatever you want to search for, you should just type skarn ahead of it. So you want to go for train test split you just type skarn skarn stands for scikitle learn almost almost all machine learning algorithms are dumped into this library almost every ML almost every 95% of the machine learning algorithms that you use you'll see it inside scikitlearn I click on that you get the official documentation Click on it and I'm first of all going to come over here and I say train test net. How to use this? So if you see carefully it says inside the skarn there is the subm module which is called as model selection which has the train test split class. So I say from skarn domodel selection import the class train test split from skarn domodel selection import train test split which is what we did so far. It says dian you give me x and y that you have already divided because this x has to be further divided into two parts. Y also has to be further divided. So you give me X and Y and I say yeah sure this is the X and this is the Y given. Now logically after giving X and Y but see X and Y has to be divided but 25 rows are there it asked me Dan you want to divide it into what percentage? I decided 8020 for now. We can change it. I decided 8020. So that's the test size parameter. How much percentage you want in the test? You can give it over here. So I want 20% of the data. So I come and I say test size is equal to 2. I don't give train size. No. If train size is also none, it will be 25. If I only give test size and if I don't give train size, it will be automatically computed. It's all written in this documentation. But this is just I'm explaining you. Okay. Then I want to tell you the shuffle parameter is by default true. What happens after it is done? I say you know this is not how it is done. I actually wrote it here. This is where. So it gives me four parts. X train X test. X train X test. Y train Y test. You cannot change the order. You cannot change the order. And this is what you get. You just have to supply train test split X and Y into this. And then I would like to print the shapes of each. So extreme shape, extra shape, white shape, whites shape. So obviously 25 rows are there because you said 80 20 it is splitted as 20 and five. Quite obvious because you said 80 triangle is perfectly splitted like this. We understand it. Okay. So, 20 rows are here. Five rows are here. Same is over here. You can see these are 2D and this is 1D. That was quite as expected. Okay. Now, now let me actually look into X train. So do you all agree that ideally extra will be having the data because because we have just seen when you look into the data frame this is rows 0 1 2 3 up to 24. So obviously when I'm dividing it in such a way that the first 20 rows will be here and last five would be here. Do you all agree that row number 0 to row number 19 will be here and row number 20 to 24 would be here? So I repeat 0 to 19 in the train and 20 to 24 in the test. Do you agree or not? Because I said 80/20 split ideally yes but that does not happen and that does not happen for the very obvious reason that the shuffle is true. shuffle is true. So when I do it, you see over here you are not having it starting from zero because our shuffle is true. So one way you can say is dash what we can do is we can make shuffle as false. If I make shuffle as false then it would be perfect. So if I come over here and I say shuffle as false. I do that and this is what you see. But this is not a good idea. This is not a good idea. You should never make shuffle as false. Why am I saying that you should never make shuffle as false? The reason I'm saying you should never make shuffle as false is because of some scenarios like this. Consider this scenario. If you see carefully, we have brand and the price of the phone. Apple phone costing this, Apple phone costing this, so on and so forth. We have 11 records of Apple phone. Actually 10 10 records of Apple phone. Then after 10 records of Apple phone, I see there are around 10 records of Oppo phone. After that I see there are around 10 records of Samsung phone. After that there are 10 records of Vivo phone. So if you see carefully right now the scenario is something like that. So I have 10 records of Apple phone. Then I have 10 records of Samsung phone. Then 10 records of some other phone. Then again 10 records of some other phone and again 10 records of some other phone. This is essentially what is the information I have. Now in this scenario if I'm dividing the data into 8020 then we know that this much part will be going into the training and this much part will be going into the testing. Do you all agree that this will be the training part and this will be the testing part? Yes. So you know what are we doing? The machine learning model is getting a chance to study Apple phones and their prices, Samsung phone and their prices, OPO phone and their prices, Vivo phone and their prices. But it never got a chance to study. Let's say these are the Xiaomi phones. Why? Because the data was in group. So it is like you are asking it to study Apple, Samsung, OPPO, Vivo and predict it for Xiaomi which is again a garbage input because it never studied and the machine learning fun is very clear garbage in garbage out. If you have never given a single sample of Xiaomi phone to study, how do you expect it to predict correctly if there is no historical data? And this problem is happening because the data is available in groups. Are you all understanding why shuffle is important? Yes, because we really don't know in real life whether the data will be in groups or not. So shuffle by default as well they have intelligently kept as true only and that's always good. You should only not shuffle when you are actually working on a time series problem. Else in all other cases it is advisable that you shuffle the data. Else in all other cases it is advisable that you shuffle the data. So now we understood that we have to keep the shuffle as true. Now the major concern when I run this I can see here I have 1159 onwards when I run it again my shuffle is different when I run it again that means every time I'm going to run my shuffle will be different so later on after this when I create a machine learning model at that time also the model will be studying different data every time. So the overall results of the model will also be different. That is why the problem. So how do I ensure that all of us are having the same shuffle? I want to ensure that you and me, everyone is having the same shuffle always. How to make that possible? And for that they have something which is called as a random state. Random state that's the only documentation available about it. It is used to control the shuffling applied before the data is split. you should pass an integer value over here for reproducible output across multiple function calls. So random state can be given as any integer value. So random state is used to ensure that the split is reproducible. So if you run your code multiple times you'll get the same train test split then can be any integer value in the range 0 to 2^ 32 -1 that is this number also I would say some of the common values which most of the people end up giving are this 0 1 42 100 1 2 3 4 although you can give any number but try out these only they are good in most of the cases so now what is the advantage I gave the random state as 42 to see now I'm running and now I'm running this I got 931 12 to5 as the row I run it again and I run this again it doesn't change. See this is cell number 34. This is 35. I run it again. This is 36. I run it again. 37. But this doesn't change. Run again. Run again. This does not change. This does not change. This does not change. So that is what is the magic. and and and this is not only applicable to my Jupiter notebook friends. We can see that uh this thing is perfectly being found out over here. So we say uh oh I wanted to say yeah this is done. So right now the scenario is you had this data you divided it into X and Y and then you so I was saying we had this data we divided it into X and Y. Now you can see how your X train is and your X test your X test will also be same as mine. You can check it out 100% it will be same. So our train and test are the same. Now it is the time for us to go and create a machine learning model. So the model which I am going to create is linear regression. So I go and type linear regression skarn. See this sklearn is very important. This skarn is very important. Right? You can see every time I have been writing the same thing. So here I say enter uh malopath. So getting into this now I'll copy this. I'll come over here and I say now it is the time to go for model training. I'll be using linear regression from scikitlearn. And this is where you find it from sklearn.linear model. I repeat from sklearn.linear model you import linear regression. So you create an object sorry you you just import the class. So in skarns module called as linear model you have this class. Then you create an object of linear regression which is this. You can name this object as anything. Object- oriented programming that we studied during Python. You import the class. First time you may find it little challenging to remember how to remember from linear model. But because it is linear regression, they have dumped it into a sub package which is called as linear model. So from sklearn.linear linear model import linear regression and then I'm creating an object of linear regression. So as we can understand I am importing the linear regression class. Then I am creating an object or instance of that class. Then I say to the object that okay now you are ready go and understand the relationship between X train and Y train. Go and understand the relationship between X train and Y train. So I say linear regression go and fit fit the model. Fit means study study the relationship between X train and Y train and it says my study is over. It says my study is over. You can see it says fitted. If you try to understand what it has understood, it doesn't show you anything. It's it's just the intelligence captured which is the line of best fit. Now if you see we just understood that see there are following parameters of this algorithm. There are following parameters. Right now we are ignoring it. And each and every parameter is explained here. Fit intercept explained. Then copy X tolerance and jobs positive. Copy X tolerance end job positive. And after parameters you have attributes. So coefficient is nothing but the slope and intercept is the intercept. So obviously we know that When pitting is done, we get a line of best fit which will be having equation y = mx + c where m is this and this. that is what is available in the coef and intercept parameter of the linear regression class. So if I show you this see linear regression if you don't put the square bracket zero you still get the output lr dot why lr lr is the name of the object lr dot coef which gives me the slope which is the value of m and this is the value of c but it is the list inside which I'm interested in the number So I will pick up the value at index location zero. Okay. So I say therefore the equation of the line is y is equal to this. Let's consider for example if there is a student if there is a student who has studied for 5 hours then I want to know what is the score that he is likely to get. So that is essentially what I'm trying to do. So what is the y that you are likely to get? So that is going to be m into x + c. M and c values are already plugged in because m and c are there I plugged in. So m into 5 + c that happened because m is this and this is this and you get 51. So if he studies for 5 hours he gets 51 as the score 51 something that is what is going to be the score. And also one more thing over here you know here m value is 9. What does this mean? This means that for every one extra hour that the student studies his score increases by 9 68. This means that every additional hour studied the score approximately increases by 5.6 by 9.6 and the intercept is the score when the number of hours studied is zero. So if a student does not study at all still he will get 2.48 marks. This is what it says. In fact, if you look into this line, it says that if the student does not study at all, it says that still the student is likely to get 2.48 marks. 2.48. Are you all understanding? If he doesn't even study 2.48 to That's confirmed. That's confirmed, right? And after that for every 1 hour of extra study because the M is 9.68 for every 1 hour of extra study his score will increase by 9.68. And if that is the case let's make this as six. You can see over here if I take this number and add 9.68 it is the same. So for every 1 hour of extra study the score is increasing by 9.68. Yes. So that's an interesting observation that we see. Now now that you understand this but come on that would be crazy if I'm going to compare it this way. If you remember, we had ideally earlier printed the shape of train and test set. You remember this? So here I already have a test data set of five rows. Let me use that for prediction. So my linear regression model is going to predict on X test and I will call it as Y prediction on the test as I discussed it yesterday. And I can also create the Y prediction on the train right by predicting on the training and the testing data set. So that is here. Now let us look at the values. So I want to look at the y test and y predicted test. I want to look at them side by side. So I'll be using the zip function of Python. So zip function will be stitching them together. This value, this value, this value, this value, this value, this value side by side, one below each other. And for you to view the results of the zip you need to type cast it to list. So actual was 81 predicted as this. Actual was 81 predicted is this. Similarly for the other three 217. So you see they are not exactly the same. Why they are not exactly the same? Because it is going to predict on the basis of line of best fit. So if you look into this person, this student studies for these many hours and he gets a score for this. But if you ask this to the machine learning model, what score will he get? The machine learning model will be like okay studies for these many hours. It will go on to the line of best fit and this is what it will predict. So essentially this distance is the error. This is the actual this is the predicted. So this is the error. But this error overall must be small. You should not introspect it just with respect to one point but overall it must be small. So how do you quantify the overall error in the model? You have metrics. You have metrics which can help you to quantify what is the overall error in this. How are they called? How are they calculated? We will see that after the break. Right now I'll directly show you that. So these metrics comes in the evaluation phase of the model. And I am going to evaluate the model using the R2 score. So sklearn has a submodule matrix where I import the R2 score. I calculate the R2 score and see how convenient it is. It says that tell me the actual and the predicted. I'll calculate the R2 on train. Similarly on the test tell me the actual and predicted I'll calculate the R2 on test. 94 on train and surprisingly on test data set it is even more haha but but but but but do you remember yesterday we discussed what is a good model what did I say what is a good model the one which on the R2 has minimum what accuracy a good model should have minimum 80% and the difference between train and test must be less than equal to 5% So here all these conditions are satisfied. It's a very very simple and obvious use case. I was expecting this to be higher and this to be lower. That's not ideally a good sign. But the reason you are getting that is we are having a very small toy data set. How do you expect it to give awesome results? But still it has given quite good. But still it has given quite good. And you can see over here that's the overall pipeline train and test train and test. And this does not actually end the project. We are act we have just started. This is not the only metric. There are many other metrics. But to start with, yes, for now, yes, that's the metric. And as you know, right, I want you to quickly try this out. I want you to quickly try this out. Run it and tell me if you are also getting the same result. Exactly the same. You have to run it directly from the collab only. You ran it, Raul. So fast. How come you typed it so fast? I am actually surprised to hear what's your strategy. I can see same same same. Very nice. Many people rewarded. I can scroll and check it. I see around 15 reward. What if what if now I come down see what can I change? What can I change? I can maybe try changing the split ratio. Now my data is very small. So I will not change the test size. But I will try changing the random state. Let's say I'll change the random state as zero. You know why am I doing that? Because I don't want I don't want my model to give higher less training accuracy and more testing accuracy. So I'm running all and yeah this is awesome. Random state zero and my problem is solved. You also try putting random state value in your code. You also try going over here and try putting the random state value zero in your code area and try it out. We are getting good results. Okay. Is there a better way further? Right. If I if I say with random state zero I am trying out the because the data is small. So 90 and 10 ratio 90 and 10. So the latest results are this. I'll just uh paste it and let me run on. Am I getting better? 95 92. No, 9010 is not. This is better. No. Okay. So, I will try out maybe a ratio of 7525. 7525 is also not that great. Let me try 7030. 7030. 94 95. Not that great. So, I think the best was this only 8020 with random state zero. Restart run all this is the best one so far but there are some other metrics what can be the different value of random states now again if I say something you will find it offensive manu just like last time you found it and over there you make me only responsible right so because you are not attentive in the class you held the instructor responsible responsible for the things that's exactly what you did last time if you recolct right you wrote that all things in the feedback as well to tarnish my image that's why I was a little upset on you in the previous course you might have noticed that coming to your question manu random state can take any number between 0 to 22 -1 although Some of the common values are this though you have an almost infinite range to try but if you go to Kaggle or any Google blogs or any YouTube or anywhere these are the very common values which people try out but of course I mean that doesn't limit our thing but there should be more than something 42 43 100 2 very common values whatever Whatever it is, whatever it is, there are no guidelines around it. The same thing applies for everything. Whatever thing I'm telling you is for the real time things only. I'm not here to talk about how it will happen on an academic project. The only difference over here is the data set is small but the tips are all for the realtime settings only. Cool. Okay. See now the metric that we use hey wait did we did we study the metrics like mean error mean absolute error etc earlier during our earlier session all of a sudden I recalled did we study the metrics like mean error mean absolute error root mean square error and all those things in our earlier class earlier uh subject so let's understand it now this will take almost half an hour to understand all the metrics. Now see we are going to understand what all are the metrics which are available for solving a regression problem. The first one is R2 score but I am not discussing it right now. I am starting with the very first one which is called as the mean error. Mean error is calculated as summation i = 1 to n of y actual - y i predicted whole upon n. So let's consider a scenario. So if I have let's say line of best fit inside that let's say these are the four points. So this is the actual point one but the prediction will be here. This is the actual point 2 but the prediction would be here. That's the actual.3. That will be the prediction. That's the actual point 4. And this is the prediction. Let's assume that this is four. This is four. Let's say this distance is -4. And this distance is minus4. So in this scenario, how do you calculate the total error? So the mean error can be computed as 4 + 4 + -4 + -4 because four points are there which is going to be 0 upon 4 which is zero. Yes, we can actually say that the mean error over here is zero. Let me consider one more example. If I consider one more example uh maybe like this. So let's assume that in this case the actual is over here and the predicted is also over here. In this case the actual is over here. predicted is also here. The actual is over here. Predicted is also here and the actual is here and the predicted is also here which means mean error will be actual minus predicted. Actual minus predicted. Actual minus predicted. Actual minus predicted. Actual minus predicted. That was total. So here actual minus predicted is 0 0 0 upon 4 which is 0 upon 4 which is 0 only and that's absolutely correct. But something which I don't find very very convincing in this scenario is on this line if I say the error is zero I can accept it positively. On this line if someone says the error is zero I am not ready to accept it. I am not ready to accept it. See if you logically see is even a single actual point on the predicted are the actual and predicted same even one time. No. So how can the error be zero? In fact every time I see a constant error 4 4 -4 -4 so this is kind of deceiving us. This is fooling us. Do you agree that this is not the ideal right metric? This is fooling us. And that was the problem with the mean error. So the very first metric which was invented was mean error only. But then slowly researchers realized this as a problem. So to solve this problem which they see in the mean error they came up with something which is called as mean absolute error. So the formula for mean error was very simple. I = 1 2 N Y actual - Y I predicted whole upon n. Now the only difference is you will find the absolute of this. Earlier you're doing actual subtraction but now you're finding absolute. So if I consider this first example for mean absolute error absolute of 4 absolute of 4 absolute of -4 absolute of -4 whole thing / 4 which is 16 upon 4 which is four that's convincing and here does it work or here it fails absolute of zero absolute Absolute of 0 absolute of 0 absolute of 0 upon 4 which is 0 upon 4 which is 0. Now that's good. So this is good. So mean absolute error is definitely better than the mean error. Are you all convinced? Are you all convinced that mean absolute error is better than the mean error? So that's my example A. That's my example B. I want to take one more example C. So here what I do in example C is I want to consider one more line. The actual and the predicted are having a distance of +1 and + 7 respectively. and on the negative side I'm assuming this distance to be -6 and minus2 respectively. So therefore in this case you can actually say if I have to calculate the mean absolute error in this scenario absolute of 7 absolute of 1 absolute of -6 absolute of -2 whole upon 4 7 8 9 10 16 which is four. Now I am not happy with this. Now you might be thinking that now what is the problem? Everything looks good. No, everything is not looking good. I'll tell you why. If you look into the example number B, it is saying that the total error is four. If you look into example number C, it says that the total error is four. Now if you ideally see if you ideally see and observe the lowest or the highest value over here is +4 the lowest value over here is -4 and if you see over here the highest value over here is + 7 and the lowest value is -6. So if I kind of put this over here I can actually say so do you see that the distribution of the error of blue box is higher than the pink box. So it is obvious that the error in the second example should be higher because the spread of the error the variance is wider and therefore the mean absolute error is expected to be higher for the next example but it is not. Are you all convinced that is exactly where we see the drawback right? Yes. So what I do over here is fourth one. I propose to you the next one which is because normal is not working. We square uh we absoluted because absolute is not working. We are going to go for the mean squared error which is summation i = 1 to n y actual minus y high predicted whole squared upon n which means if I'm talking with respect to example b my mean squared And error will be 4 4 - 4 - 4 4² 4² Take out your calculators. I need your help. Tell me the final answer. But now you'll definitely need calculator. 7 1 - 6 - 2. This is H 22.5 perfect now we clearly see that this is also perfect and we see this is also perfect but there is I see one small issue in the MSE E if you look into the example the lowest value is minus4 and the highest value is + 4. However, because we are squaring every value, the total error has gone exponentially up to + 16. This looks little intimidating and sounds like the model has a very huge error which ideally is not because we squared the results have gone exponentially higher. the same can be seen for example. See, therefore, now I propose to you the next metric called as root mean squared error. Root mean squared error. And the idea to calculate that is very simple. It is nothing but calculate mean squared error and obviously go for the square root of it. Simple as that. So if I talk about example B and if I talk about example C. Example V R M S E will be square roo<unk> of 16 which will be 4. Second one you will tell me calculator experts 4.7. Yes. Now all these values are quite well in range. That's perfect. So so root mean square error is one of the best metric. What is the what is the best value of what is the best value of RMSSE that I can get for any machine learning problem? What is the best value? Best value. Best value guess. Ah zero, right? Zero. Sometimes you try changing your random state, you try changing your train test split, you try changing the machine learning algorithm and many other things. But you know the value of RMSSE that you get yes I'm not bluffing but the value of RMSSE that you get is 12,000 11,230 to 10,000 something. Sometimes you get 14,000, sometimes you get 19,873. Now in such cases you may have a feel like oh my god I'm trying so many things. When Darian showed us an example he showed us the values come as small as 4.744 but I am getting so big values even after taking the square root that means I am doing completely wrong and the answer is not. So you may experience such things as well. So I propose to you the next metric which is called as R2. I propose to you the next metric which is called as R2. R2 stands for R2 only residual square. So let's talk about what is we can say now what can we do is let's say no I'm just looking for some smart solution and yes I'll share the link of this video it's good everything is nicely written why should I unnecessary invest a lot of energy in writing and let me focus focus on the explanation. So what is R2? R2 tells listen to me very carefully then this line will make sense to you. R2 tells how much important is an independent variable for predicting the dependent variable like for predicting the sales of ice cream for predicting the sales of ice cream. Do you agree that temperature is very important? 100% you agree to that temperature is very important for predicting the sales of ice cream. Yes. Now, what is the usual qualification of the person who sells the ice cream? What is the usual qualification of the person who sells the ice cream on the street? Ice cream lelo ice cream lelo with that right maybe not even educated below graduation maybe primary school yes now I tell to that ice cream wala bha move I am a PhD highest qualification I am the person with the highest qualification no one has more qualified than me you move I will sell the ice cream So will you come to me for buying the ice cream because I'm a PhD holder and you will not go to the person who is just a primary primary past person or primary fail. Will you consider the oh daran is PhD holder let me buy ice cream from him and not from that guy because he's not even educated. Do you take a decision like this while buying your ice cream? No. So can I say can I say that the education level is not a very important independent variable for predicting the sales of ice cream right? Yes, that's exactly what we have to understand over here. R² R² it represents the proportion of variance of the dependent variable that has been explained by the independent variable. How much variance of the dependent variable is explained by the independent variable is R². In simple words, it is the goodness of fit. So it gives a measure of how well unseen samples are likely to be predicted by the model through the proportion of explained variance. The maximum value of R2 is 1 and the minimum value is zero. So if you are getting for some reason R2 of zero that means the independent variable is just not important for predicting Y zero important and if I am getting the R2 as 0.80 80 that means that independent variable is responsible for causing 80% variance in the dependent variable or 80% effect in the dependent variable. That's the formula for calculating R2. We will be exploring that also. Right now you can just have a look at it. It says Y actual minus Y predicted. This is like AI and PI which I was writing actual and predictor. This is again actual and this is the mean of actual. So let's say there are 10 values in the actual. The average of that is Y bar. It's not Y hat, it's Y bar mean. It's the mean of actual. That is what we see in over here. Let's go ahead. Now I say let's look into the example. So if I say my R2 is 80 that means 80% of the increase in the ice cream sales is due to increase in temperature. So let's add another useless independent variable which is the education level of the worker. And when I add education level of the worker over here the R2 increases from 80% to 85% indicating that if Dan goes and sells the ice cream then the ice cream sales will increase even more. then the ice cream sales will increase even more and that is what is the thing now is that correct? So that is the drawback of R2. So what they say is here this is one limitation that it increases by every independent variable added to the model that's misleading. Sometimes variables might be useless with minimal significance. So adjusted R2 overcomes this issue by adding a penalty if we make an attempt to add independent variable that does not improve the model. Adjusted R2 overcomes this. Now how is that formulated? That is formulated again with the help of some statisticians who proved this. We need not get into why but it is nothing but adjusted version of R2. So if you see the formula 1 - 1 - r2 which we calculated earlier n is the number of rows n - k - 1 k is the number of dep independent variables number of rows number of independent variables that is what we have. Now the advantage of adjusted R2 is if you are adding useful independent variable adjusted R2 increases. If you're adding useless independent variable adjusted R2 decreases. Exactly. Yes. And that means there is a bias getting added because of the R2 and hence adjusted R2 is what has to be recommended. That's absolutely yes. So now that you understand this, so let us now get into this. So what will I do is copy this link and come over here. Perfect. So that's the link that we have in over here for some reason. If the link is not clickable, I'm also adding the name of the video. I don't know in case if in this if the link is not clickable for some reason, you can actually search it. Now I'll do it here only for you. Let us consider the following things. So extract the values of the table so that I can directly copy paste it into Excel sheet. So now this is what I have. Awesome. Right. So that is very good in these all scenarios. So this is what is the thing. Now I need to calculate the mean error. Now see this is TV. What is TV? TV is nothing but independent variable one. This is the independent variable two. This is independent variable three. This is the dependent variable and this is the y predicted. This is nothing but the y predicted. So this is what we have. Now what we have to do? We have to calculate the mean error. How do you calculate the mean error? The way you calculate the mean error is over here by using the formula which I showed you over here. Y actual minus y predicted upon n. So how do I calculate n? Well, let me write down over here. What is the value of n? n is nothing but the count of this. Five values are there. So n is five. I'm just computing it from a formula. n is five. This is n computed. Okay. So, y actual minus y predicted. So, here I say I want to compute y - yhat. How do you do that? Equal to 22 minus this. And in this way, we completed all of these values. And what is going to be the mean error? That will be the total of this. So what is the total of this? So total of this will be sum of this f_sub_2 to f_sub_6. That's correct. And mean error will be is equal to this divided by n. So the mean error in this actual versus predicted is 0.02. Did you all understand? I'm using the same formula which I taught you just a couple of moments back because we are going to work on this case study as the next problem statement. This is the actual case study consisting of 200 rows where you have not one independent variables but multiple independent variables. Then we have to calculate the mean absolute error. So how for that I need to calculate absolute of y minus yhat. So y - yhat is already computed. I will have to calculate absolute. So use the absolute of this equal to absolute of this. And then I'm going to take a dump. Then here it's going to be is equal to the sum. So sum of these all values and this is is equal to this divided by so the mean absolute error in this case ko to be 1.4 4 to 7.1 is the total of the absolute divide by 5 is the value of n. Tell me if you understood it or not. It is same thing which I showed you. I have just converted it into the excel file. Then we have to calculate mean squared error. So for mean squared error we have to first calculate the square of it. So I say it's going to be this multiplied by this. That's exactly what I want to do in the squaring. So this is going to be is equal to this multiplied by this only. And that's how I am getting all of these values. Here is where I will say I want to find the sum and here is where I say this pi and this is what we call over here. So this is the mean error. This is the mean absolute error. This is the mean squared error. And obviously in the same way I can calculate the RMSSE which is equal to square root of this and this is the RMSSE. So that's essentially what we understand. So all this metrics are populated so quickly. Are you all with me? Did you understand how all of these metrics are computed right up to MSE? And if I went wrong anywhere in the calculation, do let me know. So the RMC on this model is 1.68. We know the formulas for it. So that's not a problem. We have to go for formula for R2 scope which is can be easily represented like this. That's the formula. Now let's get into the calculation of R2. So what all things we need for R2? Let's keep it accessible. I need Y actual minus Y predicted square. So Y actual minus Y predicted is there. Square is also there. So I think numerator is there. Numerator total is already there with us. Am I correct? The numerator total which is 2.83. So can I say this numerator is 2.838. This is the same y actual minus y predicted 2.838. Right sesh that was very fast way more faster. Please slow down. Okay no problem. So now you can take a break by the time I'm done. So this is what we got and then now it is time for us to compute y actual minus y bar. I need to calculate y bar. y bar is nothing but the average of this. So my y bar is going to be is equal to average oop sorry is equal to average of yeah that's correct 18.4. Now what will I have to do? y actual minus y bar that's the first thing that I have to do so is equal to y actual is 22 minus y bar is this that's correct but as in when I translate it down right so here I got the value but next time when I scroll this down this next value it will select is 10 and after that the next value it select will be blank blank. So this time it can go wrong. Did it do the referencing properly? No. So my Y bar should not change. I have to lock this value. What is this? This is D7. So I will have to put a dollar over here to lock this value right. Yes, you know that, right? So every time it is 22 - 18.4 4 is this 10 - 18.4 is this 15 - 18.4 is this. So it's essentially this minus this 3.68 this minus this this one this minus this this one this minus this this one this minus this this one. So we have got that but what we want we want the square total. So I will have to calculate the square of this. So this star this I have to calculate the square of this. So equal to this multiplied by this and then I find the square and what I have to do I have to find the total of the square. So total of that is going to be sum of these all values. So see this is my numerator. This is my denominator. I will specifically let's say because I want to calculate it over here. I'll put it over here. I don't know why it's not moving. All right. Ticked or what? No problem. I'll write here only. I can at least scroll it. Right. So I have got my numerator. I have got my denominator. What is my numerator? This is the numerator 2.83. So I say is equal to this. Just for convenience. What is my denominator? My denominator is this. Then I say my R2 how is that calculated? is equal to 1us numerator upon denominator numerator divide by denominator and it is 0.97 for this model did you go behind now but I think it's correct everyone followed this everyone followed this right Yes. So, R2 is also computed. Now, we have to compute adjusted R2. That's the last formula. That's a formula. Right click, copy image. And you know these metrics are not only for linear regression. Any regression algorithm you study, the same metrics are available. That is the reason we are spending so much time on it. Any machine learning algorithm which is of regression type will demand the use of the same thing. Yes. So now to calculate adjusted R2, we need R2 only. So here once again let us calculate the numerator and we will also calculate the denominator. So what is numerator? numerator is equal to 1 - r2 which is this multiplied by n -1 what is n is this 5 n is 5 - 1 so we got this so I just applied 1 - r2 which is this into n - 1 n is 5 which we had already saved here I also have to find p is the number of independent variables. So I know there are three independent variables. So I will just count this now. And count of this. Oh, it's count has to be only done on numbers, right? So I can say count this three and yes, let us check it. No problem. 1 - R2 1 - R2 R2 is nothing but L15 into into into is that into yes B into what N minus one N is 5 - one looks like correct only numerator okay Sure numerator is oh yes total not this that's correct that's correct so 89 should be the value that's correct I selected this by mistake this is the total here we have the same numerator how do you compute the denominator n - p -1. So what is n 5 - - p is this -1 n - p - one and this is what I have. So adjusted R2 is equal to 1us numerator divided by denominator which is 0.5 and I think again there is something wrong which I have done over here someone please check or it's correct you have the access to the sheet here I say mean absolute error mean squared error is what we are trying to locate into. So now I say from skarn dotmetric import mean absolute error mean squared error. This is exactly like this from sklearn dot matrix import from sklearn dotmetric import these two here it was R2 so we imported this then I am going to say the mean absolute error on train is give the actual and predicted the mean absolute error on test give the actual and predicted the mean squared error on train give the actual and predicted. The mean squared error on test the actual and predicted. Then the root mean square error on train is square root of RMSSE of train is square root is this and this is the one. And then let us print the evaluation metric. This is the wall and then in between I want to add a separator then RMS no mean squared error and then root mean square error. So you can see over here the mean absolute error mean squared error and the root mean square error. You don't have to actually calculate any of these values. I showed you everything in the raw way. But you now know internally what is happening. Can I assume that you have that confidence in how these metrics are coming? Right? It's not magic. There is a math behind it. That's it. And we know among this the metric that you will finally refer will be only RMS. You will not go to MAE and all. They have drawbacks. you only refer RMSSE and the R2 or adjusted R2. So now in this we saw that the R2 is this. This is also what we have studied. I want to evaluate it by calculating adjusted R2. Now we know what is adjusted R2 and this is the formula for calculating it. We have already got that. So the first thing that we need is the number of rows n and we also need the number of predictors. So how many number of rows are there in the training data? 25 rows were there. We divided it into 8020 split. So training data how many were there in 80/20 split? 20 rows right? Yes. 20 rows and testing five. That's exactly how we divided 20 and five. So I will be having to calculate adjusted R2 on train and then on test. So for that my n is going to be extra do.shape of zero which is the number of rows and this is the number of columns and this is how I calculate it. And similarly after this now uh is that correct? Let me just see. Now my n is going to be for x test. Of course my p remains the same but I will still take it from x test and I apply the same formula. You don't have any direct formula over here. Remember you don't have any direct formula over here. So This is the raw logic that you have to implement. So 95 became 94.88 94.54 became 92. Still it is a good model and we can actually say ending to this. So as a node I say in industry the trusted metrics for evaluating the model are R2 score adjusted R2 score and the third one is the root mean square error. So I would still recommend you to go ahead with these three. Try out all. I mean anyways R2 is by default calculated. So it is recommended that you try out everything. It is recommended that you try out everything. As simple as that. We are going to get started with a multiple linear regression problem. So what is good R2 or bad R2? Uh see we discussed that also. Uh if you remember uh any metric any metric uh here the same rule applies over here absolutely same a good metric even if you consider for R2 test R2 must be at least 80% and the train minus tests must be less than 5%. In classification when you get an accuracy score also there also the same rule applies. So this is the bare minimum. Sometimes you don't even get this in banking domain. You know getting 70 72 itself is considered as a big achievement. We will discuss all those scenarios but yes I mean on a very higher side this is what is accepted. We are talking about this data set. So now we have three independent variables TV, radio, newspaper and there is one dependent variable which is the sales of it. So if you want to download this data I am sharing this link with you. You will just right click on this save as and so we have this data and let me also put in to our D3 folder. file not wrong surprising things I have folded location things yeah it's uploaded to our D3 folder as well and now let's get started so I'm creating a node group now multiple regression using this and how many are 50 in the class. Okay. Okay. Now let's get started with this. So this is going to be the problem statement. So then I'll explain you what all things we have in the data set. So number one is importing the necessary libraries. So I import numpy pandas. Then I import mattplot lip input seb also input warnings and I say warnings dot filter warnings. So this is going to ignore the warnings. Then my data frame is equal to PD dot read CSV and the name of the data set is advertising CSV and I say loading the data assuming it is in the same directory and let's look at the tape of the data set. So I say over here shape of the data set is df do.tshape. So I understand that there are 200 rows and five columns and then let's look at the first few rows and I say df do head. So now first and foremost thing we see over here this is what is the data set. You see there is uh one additional column which is just not required right and uh basically basically what has happened is uh this is 0 1 2 3 which is the uh default indexes and basically the data set is such that they had created column number one which is serial number then column number two for TV then the radio then the newspaper and sales. So this is the serial number column. Now serial number is not useful for predicting the sales. So I am going to ignore it. So how? So I will first of all copy this. See this name of the column is little odd right? Don't type in such cases. Always select and copy this. Ctrl C. Done. And then I say that the column this is just an index column and we can drop it. So I say df do.c columns equal to this. I can drop it. That's one way I can drop it. Or I can drop it using the axis. So I can say DF dot drop and then I specify the column to be drop access equal to 1 in place equal to true. Anything is okay. I'll say once again let us look at the first few rows of the data set and this is what we have. Now well coming on to the problem statement. So over here I want to say that the data set contains information about the advertising budgets. So basically we have data set of 200 companies and how much how much money have they spent on advertising that is in $1,000. So let's say company 0 whatsoever be the name of the company that's not given to us. So this company Zero has spent $230.1,000 for advertising on TV, $37.8,000 for advertising on radio and similarly it has spent $69.2,000 $2,000 for advertising on newspaper and based on that they were able to get a sales increase of $22.1,000. Okay. In this way we have data of total 200 companies. So I say the data set contains information about advertising budgets and their sale. We want to predict the sales based on the advertising budget. So the thing is after this uh if I look at the column description then you have TV which is the advertising budget for TV in thousands of dollars. Radio is there, newspaper is there and you also have sales in thousands of units. So how much sales? Now here they are saying it's in thousands of units. You can also consider the sales growth, right? That's correct. So this is what we seen over here algorithm. So we are going to use the multiple linear regression algorithm to predict the sales based on the budgets for TV, radio and newspaper. So obviously I'm using the scikits learn algorithm and in this particular thing the equation is going to be sales is equal to beta or I can say c plus I don't want m1 into TV plus m_sub_2 into radio plus m3 into newspaper right this is what is going to be the function Now the broad steps to solve our problem are first we'll be importing all the necessary libraries. That's obvious loading the data set. We will explore the data set. Explore the data set is mainly to see if there are any missing values, outliers etc. and also to check if there is linear relationship between independent and dependent variable. After that we do the train test split. Right? So train test split is what we do. But you know after that actually I should say that I will do encoding of the categorical variables if any. Now you tell me looking into this data set do you have any categorical variable like gender qualification? No. So it will not there but but if it was there then the step number four is that and you remember now the dummyification that we learned in the previous course right? Yes. So that dmification so that technique gets applied in this scenario and then after that so you should always always you know whatever I'm giving you right now these steps are valid for any machine learning algorithm any any means any therefore I'm taking the efforts to write it down so you uh you you might see some people who are not following these steps so there are so many resources resources available do not follow the steps which they are saying these are the only steps which will 100% give you the right answer right so eyes closed and you can follow the steps so what some people do that also I will tell you in this notebook only but first of all let me tell you the right approach and once I'm done with the right approach I will tell you what other people's do and why you should not do that why you should not do that But not immediately right now. So then after this you do the train test split 80/20 split. Then after that you are going to get into scaling of the features. You remember now we studied minmax scaler standard scaler which is mainly for shrinking the values. Yes. Ha. So here you do the scaling of the features. The scaling will happen if there are columns which are numerical in nature. Obviously numerical columns will be there and you will have to go for scaling here I say you'll go for scaling the features but I think I need to mention it more properly rather than I'm saying just scaling the features you should say scale the independent variables only independent right will not scale the dependent variable. There are some people who scale the dependent variable as well which is wrong. Why is it wrong? When we come to that step, I will justify but that's why I'm writing down the perfect approach because many people do many different ways. So when you see internet, you will see all of these unnecessary noise. So you should always follow the steps which I have given you. Then we say after that comes the modeling to you over here apply multiple regression algorithm or any other regression any other algorithms of your choice. Then you evaluate the model and once the evaluation is done you predict the model right so you predict on the test data set actually first you predict and then you do evaluation I don't know why it's recommended that first you make predictions and then you evaluate using the metrics as we have and then you can visualize the result if necessary that's optional that's why I said if nec necessary. You can even save the model locally on your system because usually you will not share this notebook with the client. You will share the model with the client. You remember I had shown you one deployed version. So you do that and then you document the findings and conclusion because with the stakeholders you will have to share it and then share the results with the stakeholder if necessary and then we can go for deploying the model if necessary. So broadly in our course most of the times we will be going from step number one to step number nine because obviously every project we are not going to deploy. In fact deployment is not even there but I will take it. It's not even there but I will take it. So here right. So these are all are the broader steps that we do it. Then now I think we are good to get started. In fact today I'll show you the deployment as well. I think uh I did it's not the right time to show but I think you all are smart enough and you will understand it. So I don't think so that's going to be challenge because I I have seen right you are good enough to pick up the things. It will be optional. It is not there in our syllabus. Which one? Step number 14. It's not there. Uh step 14. But uh we will do it. It's like you know take it or leave it. If you don't understand, ignore it. But I'm pretty sure you will understand. And it is important, very important. Usually deployment is something which is taught once the entire machine learning syllabus is over. I'm teaching you at the start only because I'm confident you will understand and then you know any model that we create going forward you can think of deploying as well. So that's an advantage that you get over here. So our step number one two is done right. So I say step one and two are already done up. Let's get into step three. The first is we are checking for missing values. I know there are no missing values but I will just say df dot is null dot sum dot sort values. Don't forget that and there are no missing values. So the insight over here there are no missing values in the data set. That's good. Next I say let's look at the data types of every column. So I can say df dod dt types. Everything is a float data type. So there is no encoding of the categorical variable required. Next I say let's look at the information about the data set. So I see there are 200 rows in the data frame. Four columns non-null counts 200. So no missing values. All are of the float data type. Memory usage is this. That's also good. Then I say let's look at the summary statistics of the data set. So I just say df do.escribe. I don't like this print. Now it is better. Now if you look at this when you look at this data set let's say into the average investment most of the companies prefer to invest heavily for advertising on which medium and which they think is the least important for advertising. radio you remember this don't forget this you will have to do this analysis step by step for every column here there are only three but in real life there may be 100 as well so obviously I will not consider a data set which has 100 columns 200 columns instantly at the start but you will have to consider this at a larger scale always remember this So even if you look at the minimum investment, do you still observe that the minimum investment is still the highest on TV and least on radio and the maximum investment is highest on TV and the least on radio? Yes. Let us look into the sales column, right? Yes. Sales column if you see average sales is 14 and the median is 13. So more or less looks like sales column must be normally distributed. It looks like sales column must be so now the next thing is we are now trying to I I'll just make a note of this first of all. So data set content is that I don't want I see that most of the companies have higher advertisement budget for TV as compared to radio and newspaper. The average sales is this. So lowest advertising budget is for radio. Okay. This is what we understand. Now after that now I say let's look at the relationship between the features. Features are nothing but the independent variables and the dependent variables. So how is TV related to sales, radio to sales, newspaper to sales. So I am going to first of all say features is going to be I'll just make a note of all this. So I'll say df doc columns are this. So features are this and I say for feature in features for feature in features I'm going to go for SNS dot joint plot x-axis will be the feature yaxis will be the sales and data is going to be the df and kind is equal to regression so it will be plotting a regression plot I'll be giving giving it a proper title where I can give the font size as well and I will also say let me give the X label Y label and I'll also look at grid and grid I said true and alpha I'm keeping at 0.3 three plt.tight layout and plt dot show. You can see one below other. So first of all I can see the relationship between TV and sales. Look at this TV and sales. So we clearly understand that as the investment on TV increases the sales increases. Now we clearly understand as the investment on TV increases the sales increases. Okay. Then if you look into this, do you see the similar strong relationship between radio and sales as compared to TV and sales? Which is better? TV is better or radio is better if you look at relationship with the sales TV. Yes. Okay. And if you compare between radio and newspaper which is better radio is better. Huh? I mean if you look into this look at the line also look at the points also and this this looks like a dummy line. Ideally there is no linear relationship as well and a radio is better. So I will just make a note of insights from the above plots. There is a positive relationship or there is a strong positive relationship between TV and newspaper moderate between radio and it is very weak in between this. So this indicates that TV is the most significant variable for impacting the sales followed by radio and newspaper. But we have got this. So I created this very quickly. You can actually you know if you want you can also go for a pair plot. It's your choice. SNS.pair pair plot and let it come. So I selected only three features over here but I was I was planning to do this. I was planning to do this sns.pair plot of df. So I'll just open it separately. If yeah I was planning to go for this. So if you see carefully uh sorry yeah if you yeah this is the figure finally little up I'm taking this is TV versus sales radio versus sales newspaper versus sales we clearly see that now there is one very very important fun which I want to talk about very very important fun see uh I'm little bit going off the topic IC but it is related to the topic only. H I'm little bit going off the topic but it is related to the topic. I'll tell you why. Now I am discussing a concept which is called as multiolinearity. So what is multicolinearity? So I'm still with that topic only. So now multicolinity basically means that there must be a good amount of correlation between independent and dependent variable. But there must be least amount of correlation between independent and independent variable. I'll explain you what I mean. Let's consider an example. I say a simple layman example. So let's say I have some work in Pune. I stay in Mumbai. Okay. So let me start by saying that I stay in Mumbai and I have some work in Pune and hence I will be traveling to Pur. Okay, that's one thing. Now for travel I prefer to use my own car. For travel I prefer to use my own car. But I have three cars. Let's say one is my own car. Second is I can also take out my wife's car and I also have my dad's car. So I have three options and of course I have the access to any one of this car and I can take out any one of this car. So now let's assume okay that all of this I can take out any one of this car. So what I decide you tell me whether I'm smart or not. Okay. So what I decide is decision is I will take all the cards for my work. That to how that to how is My car, I will drive it myself. My wife's car, I will hire a driver to drive it. my dad's car. I will hire another driver to drive it. Now there is a question. Is my decision correct? I have three cars. I have some work in Pune. So I take out all the three cars. I will drive my own car and other two cars have been driven by driver. We all the three cars go from Mumbai to Pune. Do our work and we come back. So is my decision correct? Just a personal logical question. Imagine you to be my in my scenario and if you have to take this decision, would you take out all the three cars or do you think it's not a good idea? No. Why? What is your problem? What is your problem in me taking out all my three cars? These are my cars. What is your problem? Only one car is needed. Only one is enough. Very nice. Right. Only one is enough. Then why to take out the three cars? why to hire the driver etc etc things it is uneconomical let's say I have a lot of money what is your problem but then still I am unnecessary creating pollution on the road adding on to the traffic and the most important thing is as you all said it's not needed if I say I have to take out my entire family with me then at time one can still adjust that okay fine entire family is coming. So probably it is needed. But in this case we clearly understand that it is just something which is not needed. So now let's try to map it. So obviously the answer is no. I should have taken one car only because all the cars are similar and they will take me to the same place. Similarly in our data set we have no forget about that. So now you consider now you consider that I say now let's assume that my y is nothing but my Mumbai to Pune travel right so that's my target my target is traveling from Mumbai to Pune And then for traveling from Mumbai to Pune, I have the first independent variable which is my car. Second independent variable is my wife's car and third independent variable is this. The relationship in between all these are linear. Let's say but we realize that X1, X2, X3 are highly correlated with each other and hence we drop H we we realize they're highly correlated with each other. And this is called multiolinearity. So multiolinearity can lead to the model overfitting. That is model mugging up the data. So what do we see in this case? We have to drop. No, not drop. We will we will consider any one of the cars and drop the rest. We will consider any one of the car and drop the rest. And that's how we handle multi-olinearity. So basically what does multiolinearity say? that if independent variables are correlated. So what do you mean by independent variables are correlated? The effect which this car will have for achieving Mumbai to Pune travel, the same effect will be by this car, the same effect will be by this car. Huh? Now if I say that I am having a BMW but my wife is having Honda City and my dad is having a swift car then the things are different obviously we will say that no of course you'll take out your BMW in that scenario but if all the three cars are same assume all of them to be Honda city so that's what we have to assume right in this example don't assume that all the three cars are different so what I want to say is whatever effect your X1 is having in Y. Whatever effect your X1 is able to produce in Y, the same effect your X2 is producing in Y and the same effect your X3 is producing in Y. So essentially we understand that if all of them are producing the same effect then we can choose any one of its. So just like if you are going from your home to school and let's say you usually carry one one rough book for writing the things whatever the teacher is giving you right not subject specific then why will you carry three rough books you carry one rough book or you carry three rough books it does not have any sense because three rough books will just add on to extra weight but at a time in a school you are not going to complete with one rough book and start with second and start with third on the same day. Impossible. So it's not required. Basically you are unnecessary carrying extra luggage by carrying these two cars because the effect is the same. So they say that why are you unnecessarily carrying all the three? Carry any one of it. If you carry all the three then it is of no use. In fact, it is more uh uneconomical as you said because it's not required and it's like telling the model that okay you want to predict the sales you use TV also your radio also newspaper also assuming that all of these are having high multiolinearity so it's like why so it's like you're teaching the model same thing again and again model wants every time different relationships to learn and that is what we see over here. So this is multiolinearity and we have to check whether our data has multi-olinearity or not. And for that we will be plotting the correlation matrix. So this is the correlation matrix. It's it's nothing but what did we do? df.core. So entire correlation matrix comes. I printed the correlation matrix and this is what I visualized it here. You can see the values are also the same 0.05 0.35 you can see the same thing 0.05 01.35. So heat map. Yes. Sorry guys. So I was saying I'm sure you might have looked into this. So if you see over here which TV, radio, newspaper are the independent variables, sales is the dependent variable. Similarly here as well TV, radio, newspaper are the independent variables. Sales is the dependent variable. Independent independent should not be having relation. That's what we said, right? Independent independent should not be having a high amount of correlation. This is essentially what we say in over here. Uh sorry, yeah, here independent independent should not have any kind of correlations. So then I say so here if you see TV and newspaper independent and independent correlation is very less uh what is the value of correlation lying between what is the value of correlation lying between yes minus 1 to 1 what they say what they say is if the value of correlation I'll write it down, right? When do we say multiolarity exists? If the correlation between two independent variables is greater than 0.7 or lesser than -0.7 then we say multi-olinearity exist. This is the best way to identify if the correlation between two independent variables is greater than 0.7 or less than minus 0.7. So always remember this. So looking into this TV versus newspaper very close to zero radio versus newspaper there is a correlation which is positive but it is not above 0.7. And then this is TV versus radio. You have the newspaper correlation but it is not above 0.7. So we don't see any multiolinity. These are repeated values. I hope everyone knows that these are repeated values. Yes or no? Yes. Did I teach you how to mask the upper triangle of the heat map? Did I teach you how to map the upper triangle of the Yes. Very nice. Ch. Then no problem. I'm not masking it. Just I wanted to show you once. Very good. So I realize I say insights from the correlation matrix there is a strong correlation between TV and sales. So ah yeah that's that's what I forgot to tell you. So TV and radio are having a very high value. So there we see multiolinearity. Are you seeing that TV and radio we see multiolinearity. Agreed. Thank you for being my blind follower. Multicolinearity is always between two or more independent variables. Yes. Independent dependent. No. Between X only. In fact, in linear regression between independent and dependent variable, we want the correlation to be as high as possible. In here, what did we say in this figure? We want this between independent and dependent to be as high as possible. This is little less we were unhappy and this is even more less we were further more unhappy. So correlation between independent variables if it is high that is above 0.7 or less than 0.7 then it is multiolinearity. So multicolinearity is always between independent variables. So that is what we say node multicolinearity is always between independent variables very important. So there is a strong positive correlation between PV and sales and that is 0 78 moderate positive between radio and this and weak positive between this Then there is no multi-olarity present between TV, radio and newspaper as the correlation values are below 0.7 or I would say are between minus.7 and +.7. 7. So we don't see that. Okay. But this is what has been done so far. I just add some notes around this some technical explanation because I explained in layman terms and whatever I understand. So multiolinearity is a situation in multiple linear regression where two or more independent variables and not only here it's in every algorithm not necessarily here where the independent variables are highly correlated with each other. This means some predictors carry redundant information about the response. Now see this concept applies for every machine learning algorithm. So don't consider this problem that we are only discussing about linear regression. No, this applies to every ML algorithm. You have to check multi-olinearity is considered to be a shrap occurs in machine learning. Okay. Then when multicolinerity is present, it is difficult for the regression model to accurately estimate the relationship between predictor and the dependent variable. So why is it a problem? Because it causes instability in the coefficients. Small changes in the data set lead to large changes in the estimated coefficients. and it inflates the standard error of the coefficient making it statistically insignificant statistics. It reduces interpretability of the model because you can't really tell which predictor is truly responsible for the changes in the target variable. And the third one is it may also lead to overfitting in some cases. How is it detected? So you have correlation matrix that we use number one and I said preferably values are below plus -0.7. Second is variance inflation factor. This is what you will learn in the next case study. It's a very nice metric variance inflation factor where we expect the value of the VIF to be less than five and and VIF is just calculated. How is it calculated is a later part of the story but VIF let me tell you that it is simply calculated using your R2. So once you have the R2 VIF is nothing but 1 upon 1 - R2. We will see that in the next case study. Condition number which is the ratio of the largest to the smallest value. We will not be looking into that. That's not very important. And again values is also not important as of now at least. Let us look at some of the acceptable value. You see everywhere it is 0.7 in marketing for sales prediction TV radio newspaper spend drop or combine highly correlated channels in finance for credit scoring where you have income saving investment amount you remove or reduce all of the correlated financial indicators in manufacturing for the production rate forecast for predicting that you have machine hours labor hours power usage So you keep only the most significant one and remove the unnecessary one. In the retail for customer spend modeling based on noyalty points, purchase frequency and recency you use PCF. PCA is the uh unsupervised learning algorithm that we will be studying. I had uh talked about it earlier. And for logistics where you are predicting the delivery time prediction based on independent variables like distance, fuel usage, travel time, you expect the value to be less than this or I can combine these metrics together. So how are decisions made in practice in a realtime problem? Correlation between predictors let's say is less than 0.5. We are good. No problem. But if it's moderate right because every time I said you know between minus.7 to plus.7 is acceptable but it depends on the use case sometimes maybe you know 6 is also not acceptable so in such cases you may have to carefully inspect the business rule and take your decision decision that is 0.55 acceptable or should I go with what dashans said 7 well that rule is is a very generic rule. It can be little bit tweaked based on the application. Always remember that. But anything above 0.7 it's definitely high. So over there you may have to remove that and PCA is one such algorithm which can help you to remove the multiolinearity. VIF value as I told you it must be compulsorily below five. If it goes above five, there is a problem and high colinearity is also not advisable for interpretability. Let's consider an example for marketing spend. If TV spend and radio spend are let's say highly correlated. Let's see example then drop radio and build model using any one of it in manufacturing if there is a machine a runtime and power consumption having a very high correlation then keep machine runtime and drop this because it is the runtime that drives the production. This is very important. See logically if you have to select for a machine A between two independent variables runtime and power consumption I said I will keep runtime because runtime drives the production and not the power consumption. So when I say that I have three cars and I have to choose any one of it. I will choose the one logically what is important for me because this is what will be used for prediction. Just like in retail, customer visits and purchases are correlated, right? How many times a customer visits the website and how many times he purchases something? So, how many time am I visiting Amazon.in and how many times am I purchasing from that? The more I visit, the more I purchase. Agreed or not? The more I visit the website, the more I purchase. If I'm visiting Flipkart less then my chances of purchasing from Flipkart are also less. Yes or no friends? The more you visit a website the more are your chances of purchasing right? Yes. So if there is a multicolinearity between the visits and purchases which one will you keep? They say keep both as visits and purchases capture separate aspects of the customer behavior. So I'm saying do not remove it. I am saying do not remove it. It's a business decision. Remember it's a business decision. Multicolinearity does not affect the prediction accuracy directly but it affects the model interpretability and stability. So if your goal is forecasting you may tolerate some multicolinearity. very very important point that we learn out of example number three. But if your goal is to explain the relationships then it must be addressed. Very important point which comes out from our discussion in our data set. We don't have multiolarity but if you have it this is how you will address it. So now that we have uh done the problem of multi-olinearity I think uh I think one thing I forgot in the above one one step I forgot I just realized it split the data into training and testing. No, before that separate independent variables X and dependent variable to Y. You have to divide the data into X and Y. This is what the line number five teaches us. Divide the data into X and Y. So now what will I do is this is what I have. I will take TV, radio, newspaper into X and sales into Y. So I have already I already have this features but but if you want I'll just keep it once again. Then I say my X is going to be DF of features. My Y is going to be sales. Done. X dot head. Y dot head. Done. After doing that, the next thing is to do train test split. So I say split the data into training and testing. Now tell me what should I write from sklearn dot something import trail test split. What is that something? Very good. Yes it is the model selection. from sklearn domodel selection we import the train test split very nice and very very important point for all of us so I see over here that from sklearn domodel selection import train test split and then I can decide upon the ratio by default which ratio do I prefer in fact that's exactly what most of them prefer by default. What ratio do I prefer? 80/20. I will start with that. That is not necessarily the best but usually one of the good starting points. So I say this will automatically give me four parts. X train, X test, Y train, Y test, random state I am putting over here as 42. Shuffle is by default true. and then I will just check out the shape of each. So before that I hope you remember the shape of the data set. We had 200 rows and five columns. So if I'm doing 80/20 split, how many rows will go in the train? 160 correct 160 absolutely correct so I'm doing that and you can see 160A 3A 3 160A 40A looks like perfect then do you know the process what we are going to do so now that we have it the next step is now you focus on extreme dot head we have to scale these values. Why scaling is important? See if you look over here extra dot describe. Now in the earlier project we did not scale it. Ours versus score there also scaling is important. We did not scale it but we should have scaled it. I purposely did not do that because I wanted to keep it simple and anyways it was a toy data set problem so it was uh simple only. Now here if you look into this we understood that which is the most important variable for prediction. Sorry, not prediction. Which is the most invested variable by the company? TV. And which is the least one? Radio. Right now there is something which I have to say something which is very important. Just like you understood that TV is the most important world. I'll reduce the least important one by looking into the minimum, maximum or maybe even the mean that just like you understood this in the same way the machine also understand this. Okay. Now what is the problem? Just like you said TV is most important and radio is least important. Machine also tries to infer something like this which according to me is wrong. I don't want the machine to consider TV is most important and radio is least important. Why why don't I want the machine to consider that TV is most important and radio is most important is consider consider how many students are there in our class 51 are there right so I have a class of 51 students and in my class you know let's let's say this is the board where I teach this is where I stand and this is my first row. This is second row in the class where you all sit. This is the third row of the class. And let's say this is the fourth row of the class. Now I have seen I have seen regularly the following students coming to my class. So I see Priam is coming, Kamla is coming, Schwatha is coming, Manu is coming, Ramani is coming, Shanu is coming, Raj is coming. Right. Uh and in this way I see in my class regularly there are total around 48 students who are coming regularly they come regularly they be attentive in the class but let's say I see over here that in our class let's say for example there is a student called as Rohan and Reika they don't come to attend my class see whether Rohan and Raika are intelligent or not that's a later part of the story but this Rohan and Raika are not coming to attend the class so let's say I conducted total of 20 sessions in the semester this Rohan and Reika hardly attended two or maybe maximum four sessions that's it and therefore accord According to me Rohan and Reika are useless students and Anmul Kamlaka Raj Manu they are awesome students because they have attended my lecture and Rohan and Reika I don't know they they may be very good in that stuff but according to me they are useless. I I just hate those guys. I just hate and and I decide in my mind that on the day of viva I'm going to take out all my grudge which is there in my heart on them. So even if Ron and Reika answer everything nicely I will ensure that I am giving them the bare minimum marks and if they don't answer they I will fail them. I will have no mercy on them because they didn't attend my class. Huh? I have that grudge in my mind and I do the same and this all things are done just because they have not attended my class. So let's say on the day of the VA Rohar and Reika were very good much much better much much better than Manu much much better than Kamlaga but I have that grudge and I give them less marks. Is my decision justified? No right my decision is not justified. If as a teacher I have to reduce marks for Rohan and Reika, I should reduce them only in the marks allocated for the attendance of the subject. I can give them zero in attendance because they didn't attend. But in viva giving zero marks or failing them is not justified. Now consider me to be replaced by the machine learning model here because the investment on TV is the highest and on radio is the lowest average investment. The machine learning model will consider that TV is very important and radio is least important. So while it is understanding the relationship between extra and wide range if it goes on with its biased opinion like that is that justified? Answer is no. Right? It is not justified. And therefore how to make machine learning model understand that you don't be biased. You remember I explained you minmax scalar it converts it converts all the values whatever be the minimum and whatever be the maximum. Each of this minimum will become zero and each of this maximum will become one. So everything is brought into a standard range 0 to 1. Yes. Do you all remember that? Yes. So, we are not changing the values but we are proportionately squashing everything into a standard range 0 to 1. And that is exactly what we are going to apply over here. That is exactly what we apply. So now I do the scaling scaling of the independent variables. Okay. And only the independent variables. So I say over here which scaling technique will I apply? Now see I can apply minmax scaler. I can apply standard scaler right. So, so we have uh standard scalar minmax scalar versus many others. The question is which one to use when as per the industry standards. The question is also which one are the most popularly used in the industry? The question is also how should I decide which algorithm or which scaling should I use? Whenever I face a challenge figure out that. So we already know the scaling techniques. So here we say by default if you ask me one of the scaling technique which is recommended is standard scaling. By default if you ask me one of the scaling technique which is recommended is standard scaling only because it is one of the best technique to go ahead. Although there are many other techniques as well which are available. I'll just like to put up them. So if you talk about feature scaling commonly used scalers standard scaler a standard scaler will ensure that overall average of every column becomes zero and standard deviation becomes one. affectctor algorithms, linear regression, logistic regression, SVM, PCA and it is very popular. I would say the first by default is this only. Minmax is mainly helpful for neural network. So as far as our machine learning syllabus is concerned, you can kind of ignore it also but at times you can try it also. If your data is having outliers then you go for roboscaler. But obviously if you are removing the outliers then why will you go for robust scaler. So it has to be used when really needed in financial data sets where uh outliers are actually ideal values. You know removing that is not advisable. In such cases where you cannot remove the outlier because they are genuine values. It is advisable that you keep it as simple as that. So next thing max absolute scalar it will scale in the range minus1 to 1. It is mainly used in the uh text processing problem. So you are not going to use it here. Normalizer again used in text processing problem is a niche use and very less common are the contile and the power transformer. then decision which one to use. So you have to analyze the algorithm sensitivity. But you can see over here in most of the cases standard scaler will fit. So KNN algorithm, KNS algorithm, PCA algorithm they are highly sensitive to scaling. You can use any one of this. So when you have a choice in between standard and minmax standard will be the first one that you try. Then in gradient based where logistic regression and SVM are there you use standard scaler for treebased algorithm decision tree random forest XG boost scaling is not required. When we talk about that algorithm, we will discuss it. But scaling is not required. For neural networks, minmax scalar is the first one. And for sparse data, which is mainly text processing problems over there, max absolute scalar. So looking at the data characteristics also you can take a decision. So if the data is normally distributed then standard scaler is good. If it's bounded between some values, then minmax is good. If it contains outlier, robust scalar is good. If it is sparse or normally distributed, max absolute scaler or normalizer is good. And if it is having a non-normal distribution, maybe you can go for this. Also, you need to look into the impact on the business. If you are doing a fraud detection where there are high outliers, robust scalar customer segmentation is the problem. Let's say you're implementing it using the PC and K means then standard scaler for neural networks minmax scaler for financial time series problems power transformer and for IoT sensor anomaly detection minmax scalar. So which scalers are most popular in the industry? As you can see the table. So whenever you face with any challenge, you should always check the model type. Is the model distance based, tree based or gradient based? Like linear regression is distance based. Then you should assess the data for outliers. If there are many outliers then avoid minmax scalar. You check the distribution of the feature. Is it normal distribution? Is it bounded in the range 0 to 100 or 0 to 255? You run quick EDA, box plot, histogram, skewess check. You do that. That's why we said EDA is very important. You start with standard scalar as default. That's exactly what I said the most important thing. So when to use what? For most general cases, standard scalar for bounded cases which you see in the deep learning problems 0 to 255 is the usual range you go for minmax scalar. Outliers robust for text data you use the max absolute scalar. If gshian distribution you go for this and for vector normalization you can go for this. That's essentially what we understand. Then I say from scale on pre-processing import standard scaler. Then I say SC is equal to create an instance of standard scaler. Then I say extra of features. So I say standard scalar go and fit transform. So I remember I had said fit is nothing but study and transform is like apply the transformation first you study and then apply the transformation. So that is what you are doing it on extra features and on the x test you only apply the same. Why are you only applying the same over here? Y not fit which is nothing but the study on the test data set because it already has studied the training set and we want to apply the same transformation on the test data said this will help us to see how the model performs on the unseen. Now this is very very important. I'll tell you what I mean by that. So here I did that. Now what I I'll show you what I mean by that. I'm showing extra dot describe and I'll I'll also go for ext.escribe. Now if you see is the average zero for all the independent variables is the average zero for all the independent variables. It is it is it is -2.9 into 10^ -16 which means it is 0 0 0 15 * 0 followed by 29. This is 0.15 * 0 followed by 208. They are very very very very small numbers. Did you all get it? Amit figured it out. Everyone did you follow the average is zero and the standard deviation is one. That's exactly what the scaling does. Average is zero and the standard deviation is one. Okay, that is what minmax scaler does. That is what standard scaling does. And here also do you see the average very close to zero on the unseen data and standard deviation if you see over here standard deviation is close to one except for the newspaper variable. standard deviation is close to one except for the newspaper. Now you know what are we trying to figure out over here? The very important thing which I wanted to say in fact I I'll show you this effect. this effect will be more properly seen uh by showing you maybe yeah so now if you observe see what I did I did apply minmax scalar on this data set now after applying minmax scalar uh you can see the minimum value has become zero and the maximum value has become one that's Perfect. Minimum zero, maximum one. And that is perfect. On the test data set also the minimum must be zero. Now on the test data set also the minimum must be zero. But it is not. But it is not. And the maximum must be one. But it is not 98. You know why? Because when the model did fit and transform on the TV, radio, newspaper, it got to study the data and that data that data whatever it studied maybe let's consider the TV column. The TV column over there had the minimum value what let's say minimum value was 10. Maximum value was 100. Let's say the TV column had minimum value 10 and maximum value 100. So minmax scaler will do what? Minmax scaler will convert 10 to zero and 100 to 1. This is what it will do. That is with the training data set. In the testing data set, let's say the minimum value in the testing data set for the TV, the minimum value is 9 and the maximum value is 105. Possible now it's very much possible that in the test data set there may be a minimum value which was not there in the train much much lower than that or maybe sometimes it can be higher than that because it went out of the range. What do we understand? And we understand that probably this is because of the shuffle. The shuffle that we got would be a bad shuffle that it is not covering the test data set. And therefore to avoid that kind of bias we have to see that ultimately after doing the scaling as well is the train which is between 0 to 1 is the test also almost close to zero it is. And is the max also almost close to one? It is not because 98 is okay. 99 is okay. But this 1.13 is like almost 13 units above one which is not recommended. So what you what you can do is you can try you can try changing the shuffle. So I go up and instead of random status 42 let's say if I'm trying out random status zero. when I try random set as zero. My x10 is still zero and uh this is also one. Look at this. This is almost zero and this is 97 99.13. Uh-huh. So it is still.13. So I'll go up. Let me say random status one. Now if you see uh minimum is as good as zero and maximum is still unchanged. Let me say try out random state as 43 minimum is zero. Oh do you see minimum is zero maximum is this is good this is good newspaper is 63 much much out. So this value must be close to one. I say let me try out 100. See minimum is uh zero, maximum is bad, more bad shuffle. So you keep on trying out this shuffling till you get a good shuffle. So let me say let me try random status 99 more bad. So in this way you keep on trying till you get the value. So I guess the one that we tried in earlier was the best so far. Now I do one thing. Yes. So this is what we see. Minimum mean is zero and the standard deviation is as good as one. So let's say I'm going ahead with this and the scaling technique which I'm finalizing is standard scalar only. Now once this standard scalar is applied now we are good to get into the modeling part. So see essentially what I wanted to tell you is you always study and transform on the training data and you only and only transform on the testing data so that you are able to see that if the intelligence is captured in the object SC how is it performing on the unseen data also we would get into the modeling part and I'm going apply multiple linear regression model over here and here I say from sklearn.linear model import linear regression when you create an instance of this model. So as you can see we are creating an instance by the name LR. You can give any name to it as you already know it. And then I will ask the algorithm to go and study over X train Y train. Once it has done its study, I will ask it to make prediction on the training data set and the testing data set. So I say Y prediction on the train is linear regression. is go and predict on this and y prediction on the test is this then I will evaluate the performance using the matrix so let's first calculate the R2 score so mean absolute error mean R2 mean squared error R2 score is what I'm doing over here error between Y train and Y predicted trend mean square error between Y train and Y predicted trend root mean square error between Y train and Y predicted train error in between the two and for calculating adjusted R2 right so for calculating adjusted R2 I need the formula you remember I you we have already seen that right So I will calculate the number of rows in number of columns in that and then I will calculate the numerator of the train formula is this numerator of the train is 1 - r2 train into n trainus 1 denominator of the train is n minus p minus one. >> Okay. >> And that's the adjusted R2. Similarly on the test data set we do the same thing. We have already studied this. So I'm not explaining it once again. It is just application of this formula and I'm printing the training matrix. Uh I'll print the train and test set evaluation. So, MA for train and train set MA testing set MA. Similarly, MSE, RMSSE, R2 and adjusted R2. We got all and we have computed it. This image will get uh lost in the Google collab. So, this image URL is what I have pasted over here. And we can see over here MA is this RMSSE is uh this on the train it is less on the test it is little bigger R2 it's.9 and.9 and adjusted R2 is little less 89 and 89 and now that we have that this we will also be printing the coefficients of the model. So the coefficients of the model is feature- wise coefficients. So zipping together and I'll also be printing the intercept of the model. And then I'll also be printing the equation of the model y is equal to this plus this into this. >> Right? >> Let's understand this. uh before that I would just like to know after this coefficients of the model I want some new line and this equation of the line also I want a new line to be added in between so if you see uh TV is 3.76 radio is this newspaper is this intercept is this so the equation of the line is m1 x1 plus m2 x2 plus m3 x3 plus c what do we Understand? So we understand that for every one unit increase in TV, one unit increase in TV, that means $1,000 additional dollar investment, the sales increases by 3.76 units. For one unit increase in radio, the sales increase by 2.79 units. and newspaper. Oh my god. For every $1,000 additionally invested, the sales of the newspaper is increasing by only and only this much. Little bit put a stress on your memory and try to recall which was the most important variable for prediction as per us when we looked at the scatter plot TV and that can be seen over here. And what was the least important? Ah, radio. But looking at this, which is the ideal real least important newspaper. It says that if you invest 1,000 additional dollars on newspaper, you only get a sales increase of approximately six units or 6%. which when compared to TV and radio is marginal. If I have to create a model once again, which feature will I prefer to drop straight? Right. Absolutely. So now if that is the case, I say since the coefficients of TV and radio are higher as compared to newspaper, we conclude that TV and radio have much impact on the prediction as compared to newspaper. Let's visualize the prediction. Let it come then I'll talk about it. Now if you see over here what have I plotted these are the actual sales and these are the predicted sales and these are the actual sales and these are the predicted sales. So actual versus predicted is what we have plotted. So the thing over here is what do we understand from this right? So looking into the actual and predicted cell. So what what we did is Y train which is the actual and Y predicted train is the predicted. I just plotted them in blue color. Alpha is 0.5 and restful things are decoration and another one Y test versus Y predicted test rest all is decoration. Now looking at this what do we conclude? See now here if you carefully observe when you look at the left plot which is about the blue dots uh most of the points lie on the diagonal as imagine from left to the top right bottom left to the top right which clearly indicates that on the training data our model is fitting well because the predicted sales are closely matching the actual sales for most of the records because this is how it is. So this is what we see because I said color is equal to blue. A few data points deviate which show minor error but the coverage is still strong also because our R2 is high. Uh this is because of that as well. And on the right side if you see it will still follow the diagonal with a slight more towards the training data. Right? So here we understand that the model is performed reasonably well on the unseen data. That means it is able to generalize but with few cases where there are less accurate and this red is normal indicating that there is a balance in between overfitting and an underfitting model. So we see there is a slightly lower R2 over here indicating that there is a high error metric. Then overall we can understand that if the coefficients right since the coefficients of TV and radio are significantly higher than newspaper we can conclude that uh TV radio are important and newspaper is less important. So newspaper hardly contributes anything. So in business terms I can also say that the marketing managers may want to reallocate the budget from newspaper advertisements to TV and radio for better ROI. What we understand this is what we overall understand. So now what am I planning to do is I am planning to actually uh I'm planning to actually remove newspaper and see if my thing is useful right so I just added explanation about this for your better understanding but I explained the same thing So we will now try to remove the newspaper. So I say that let's remove the newspaper from the data set and see how it performs. So what will I do is I will say features is equal to only this my X and Y separated. Then after X and Y separated again I have to go for train test split. I did that. After train test split I will I'm not interested in looking at the shapes. I'm looking for scaling. So I create an instance of standard scale and I put a transform over it. Now I don't need the comments as well. That was you know it was good for the first time. Now I don't need it. Okay. Then I go to modeling. I create linear regression model. I fit over it. And after fitting I'll copy the code. after fitting till the end everything I'm copying you can see over here after fitting prediction error metric calculation this is good so far everything remains the same and this but of course the equation of the line will change. Now if you see let us look at the older results first I'll have to copy that for our better understanding we see that over here the training set having MA 1.2 testing set 1.44 means for error 2.71 3.14 let's compare with the older one earlier it was 1.2 then it is 1.2 2 only after dropping newspaper no impact but MA which was 1.44 44 actually it increased it should have reduced but it increased but let us look at the most significant metric as for us R square is 9090 latest one and the earlier one was 90 and 90 no impact adjusted R square 8990 it is 8990 only and You see that? Now I'll do one thing. You know what? Uh the reason I'm getting the same same thing is because I'm looking at only the 2 precision. I will I will prefer to go for entire result everywhere here also here and here also. like to go for the entire result that 2.2 is actually not showing us properly. So now maybe we get a good idea. So earlier it was sorry earlier it was.89 8 9 3695 from 8 942 see adjusted R2 increased a little bit on the test it was 89105 now it is 8950 see there is an improvement which means that dropping the newspaper was little for little but it was definitely helpful and these are the new coefficients this is what we understand so far from this you know how is radio actually helpful is what we understanding over here. So dropping of that was a good idea is overall what we can conclude from this. So we see over here that the equation of the line for features TV, radio and newspaper was this and Here is this. So we can see that the coefficients of PVL radio have changed slightly but the overall performance of the model is still good. So conclusion is we can conclude that these have higher newspaper available. So dropping newspaper did not affect the performance of the model. So better selection can improve the performance. Dropping newspaper did do that. Okay. And that's it. Model can be further uploaded but right now I'll not get into that. And this is what we have. H everything is done so far. And uh now after this say that let us visualize the predictions. Same code same code like before nothing new. So not really a great achievement but definitely if it has improved even this much see I need not invest on newspaper I'm getting anyways the return from other factors why will I invest on that and that is what is the conclusion so far that is what is the conclusion so now number one see uh something which I didn't explain you in the course is what I am going to explain you right now the first one is you know we are talking about assumptions of linear regression model. Our case study is done but we are talking about assumptions of linear regression model. So I'll just talk about it. So that's the topic assumptions of linear regression model. The first assumption of linear regression model is linearity. So linearity is what we have already checked. So we have already checked what is linearity. So linearity says that between independent and dependent variables there must be linear relationship. And this is what we have checked. If this assumption is violated then linear regression is not the right choice of the algorithm. And the way we check it is visualize predicted versus actual. We did that just some time back. We have done that above. Right? This is the same. But still I will like to do it over here. So I am simply plotting the visualization. So what am I saying? It says that when you do y train minus y predicted train there must be linearity. When you do y train minus y predicted train there must be linearity. So I'm just finding out the difference. I get this histogram and on the scatter plot I am plotting y predicted train versus the residuals. Right? So prediction versus the residuals and there must be linearity over here. Right. So here we say that the residuals should not show a pattern when plotting. I say the residuals should not show a pattern when plotted. So what do we understand by looking into this figure right so on the x-axis we are having the predictions and on the y-axis you are having the residuals then you can see over here that I'll jus