Linear Regression and Gradient Descent

Key Takeaways

Hello everyone. I hope you all are doing well. This side Priya Bhatia and in this video we will be talking about the linear regression mathematical intuition. I hope that before coming to this session you already have watched my previous video where I have talked about that you know what this regression problems is all about and in which category this linear regression algorithm came into picture. So today we will try to discuss a lot more in depth towards this particular algorithm and we will try to see the mathematical idea behind this. So we have observed that whenever we're talking about regression problem, what does that mean that simply we want to predict a target value where the prediction is of continuous real numbers, real values are there, right? For example, I want to predict a house. House price prediction is a great example which I think everyone must have heard of, right? Now, uh let's say that you will be dealing with a simple problem where let's say I will provide you the height of a person and I'm saying that you have to predict according to the height what is the weight or what should be the weight of that particular person. So we will be having something called as one feature which is height right? As the input value I will provide to you and the target value which we want to achieve is the weight. Okay? Now, this is something which you can say is the input feature X and this is something which you want to get the prediction as Y cap. Obviously, it's a part of a supervised machine learning where you will be given the weights and the set of records that we have. For example, let's say one to maybe 1,000 data points are given to you. Where we will be having input value as the height and the output value as the weight. Now, given this particular data set, let's say if in future I will try to give you the height or I will try to give the model the height of a person, what my what what is my expectation? My expectation is that my model will be able to give me the prediction that the weight is this much depending upon the learning which my model have done from the particular data set. So, basically what we are trying to do, we are trying to have some sort of, you know, values which is being provided to us as a as a data set. And what we will do is we will try to plot those values. So, what usually happens is let's say you will be having the data points which are given to you like this, like this, like this. Right? And then maybe you will be having another data points like this, like this, like this, like this, like this. Now, what you have to do is you need to calculate the best fit line. You need to calculate the best fit line which basically is having a minimum error. So, I will try to talk about this thing. So, let's say this is the best fit line that we will be able to get where the difference between the actual value and the predicted value is minimum. So, let's say here on the x-axis we will be having the height of a person as the input value x. And on the y-axis we will be having the weight which is the output y which we want to predict. Okay? Now, what we want is that given any height I need to create some best fit line which basically describes a line in which the error will be minimum. Now, here again and again I'm saying one thing, error. Try to understand what is the meaning of a word called as error, which you will heard as residual as well. Error or residuals. What is the meaning of this? See, whenever we are trying to do any sort of prediction. For example, in our case, our prediction will be the weight of a person given the height value. What will happen is you will be able to get some prediction, which is a Y cap. And as it's a supervised machine learning algorithm, we will be having the value of Y, which is the actual value. So, this Y cap represents a predicted value. Predicted value of a target, whichever you want to predict. For example, in our case, it's a weight. And this normal Y is something which you can say is the actual value. And the gap between these two is something which we call as an error or a residual. And what we want is in any model, our outcome should be our model should be trained in such a way that this error or this residual will always be minimum. So, what we want is we want to create one best fit line. This is what I'm saying as a best fit line, which basically makes sure that the value of the prediction minus the value of the actual should be minimum. Right? For example, let's say key I will say that this is the height. This is the point where, let's say, the height of a person is maybe 20. Now, with respect to this line, with respect to this line, if you will be able to observe with the this best fit line, this is the value of a Y cap. This is the prediction that that this is the weight. But the actual weight may be lying here. The actual weight corresponding to this may be lying here. Here. I hope you can see my point. This is something which you will say is the Y. So, for every given value, you will be having a value of Y, and you will be having a value of Y cap. What we want is that for all the observations, the residual or the error will be minimum. That is something which we want to make sure. Now you all know that whenever we're talking about a creation of a best fit line, what is the equation of a line? I hope everyone knows the equation of line will be Y is equals to MX plus C. Y is equals to MX plus C. Where M is nothing but you know is the slope of a line. This is nothing but is a Y cap where we want to do the prediction of this best fit line. This is something which you can say is the slope. And this is something which we usually say is the intercept. Intercept, right? I hope that you already have you know understood these kind of concepts in your uh very lower classes in mathematical you know part. Now here you will be easily able to observe that this kind of algorithm where the very first thing until so far we have seen that our target value is some continuous real number. Or you can say a numeric number, numeric value. In that case only we will be dealing with this kind of algorithm which is a linear regression based algorithm. Now why it is called linear regression? Because here we are trying to create a line which is linearly fitted line, right? Best fit line. So that the error will be minimum. And here if you will be able to observe the relationship between the value of X and the value of Y is somehow linear to each other. Why I am saying so? Please try to understand. If you will be able to observe as and whenever the height of a person increase, obviously the chances of uh a weight will also get right? Directly linearly proportional to each other. So, in that kind of scenario, we can easily apply a linear regression based algorithm. There are some other conditions that uh we will study in the upcoming part of the video, where I will try to talk about that, you know, in which particular circumstances you can definitely go for this linear regression algorithm. Or, in other words, I can say that what are the conditions that we have to apply linear regression algorithm. That we will discuss in the upcoming part of the videos. But, as of now, just try to understand that what is the meaning of residual? What actually want we want to achieve? And uh how basically we will be able to get the best fit line. So, here if you will be able to observe, this is the slope. This is the intercept. At any point of time, if I will be able to get the uh you know, perfect or optimal value of M and C, can I say the say that I will be able to create a best fit line? Please try to understand my statement pretty well. What I'm saying is you know that the best fit line in linear regression, what we want to achieve is a best fit line. Just now we have understood. Now, you know that the equation to calculate the best fit line or to get the best fit line is because it's a straight line, it will be Y cap is equal to MX + C. Now, what I'm saying is that please try to see this equation very very carefully. Very very carefully. What I'm saying is that in this particular equation, can I say that if I will be able to calculate the value of M and C because X is anyhow the given feature which is given to you. It's a kind of a input value, which will be entered by the user. What we want to achieve is the value of a Y cap, which is the optimal weight of that particular person as per the given data set. So, if at any point of time, I will be able to get the value of M and C, M and C, which is quite accurate, right? What we want to achieve. We want to achieve a value of MNC which is quite accurate. At that point of time, can I say that I will be able to get a perfect model or a ideal model where the error will be minimum? Again, I'm saying the meaning of a error or the meaning of a residual meaning that the value of Y cap which is what you will be able to get as a predicted value. Right? Predicted value. Minus the value of actual value which we have, right? Because we are dealing with a supervised machine learning algorithm where we will be having a actual value available with us. Okay? So, this is what we want to achieve, right? Now, the question is the complete concept which we have to understand now is how we will be able to get a quite accurate or in other words, optimal value of MNC. For that, what usually happens is initially some random value of MNC will be assigned. And we usually try to create a any kind of a line which may not be a best fit one. Then, what usually happens there are optimizers which work which plays a very important role here. So, there is very important optimizer which you should know which we call as gradient descent optimizer. Now, the task of this optimizer gradient descent optimizer. Now, the task of this optimizer is to optimize the value of MNC, meaning to get that accurate value of MNC, meaning to lower down that particular error. So, what usually happens? It's kind of a curve that we have where what we want to achieve, maybe initial case what what happens is you will be able to get a very high error. Let's say initially you will be able to get a uh you know, a random value of MNC for which you will be able to get the error as very very high. What is the error? The difference between the predicted and the actual value. Now, what will happen? What will happen? You will try to change that value of M and C and you will try to lower down the value of error. What you want to achieve as a final outcome, you want to go here. You want to go here, the lower path, which we call as a global minima, global minima. Right? In this curve, if you will be able to observe, we have a very lower value, minimum value, which we want. I want that my error should be at this point, which is approximately equal to zero. Now, you will ask me one question that, "Okay, Priya, you told us that you know, we need to change the value of M and C to reduce down the error." The question is, "How we can change that value of M and C?" So, as per the gradient descent algorithm, it says that you can easily evaluate the value of M new, new value of M as what is the old value of M that you have minus eta. Eta is a learning rate that we have into derivative of the error that you're getting with respect to the value of M. Similarly, you will be able to get the value of C new, which is equals to what is the value of C old minus eta times derivative of error with respect to the C. So, this is the way to update the value of M and C. This is the way to update the value of M and C. Now, initially, a random value of M will be assigned that you can take and take it as a old value of M or old value of C. Eta is something which is a learning rate. I will talk about this particular thing in a great detail in the upcoming part. And then, you have to calculate the difference or the I would say that derivative of the error with respect to the value of M or with respect to the C. We will talk about this also in a great detail. Okay. So, what usually happens is that with the help of this formula as of now you will try to consider that this is the way to lower down the value, to lower down the value of error. So, what happens? We will try to take care of these values and step by step we will try to update the value of M and C. At any point of time you will feel like that okay, you will be having now the error approximately equal to zero, you will stop your learning. You will stop your learning. So, at this point of time at this point of time where where the error is approximately equal to zero, what is the value of M and what is the value of C? With the help of these two values you will be able to you know create that best fit line and all the predictions, all the upcoming predictions for example, this is the line where I will be having a most optimal value of M and C. All the upcoming predictions that I have in my data set will happen on this particular best fit line. Ticket. That is the meaning of a best fit line. Now, I hope everyone will be able to understand what what I'm saying by the meaning of a best fit line. A line where we will be having a most optimal value of M and C. What is the meaning of that? Where in my curve the error will be minimum. I will be reached towards the global minima, means error is approximately equal to zero. Just now I have shown you as well, right? Just now I have shown you as well that how is this error is being operated? This is the formula. This is the algorithmical formula which you will be able to you know check it out in any of the research paper as well. It's not that we we only have only gradient descent as an optimizer, we have multiple optimizers, but this is the very first session with respect to the algorithms part that we have in machine learning. So, just try to understand this particular algorithm pretty well, okay? How basically with the help of this optimizer we are trying to optimize our value of M and C. This is something which you have to understand. Now, the question which you can ask from me uh is that what is this? How basically we are trying to evaluate the derivative of error with respect to M or derivative of error error with respect to C. Please try to pay attention again. Uh pretty much uh very consciously please try to see here. What is happening? What is error? Just now I have talked about error or residual is one of the same thing. It is the difference between the value of Y and Y cap. Y I minus Y cap whole square. Why I'm doing a whole square? Because I don't want any negative value to be there, okay? I just want to dedicate any negative value. That's why we will take it as a whole square. You can also take it as a absolute value or you can take it as a whole square. Now, this error I want to evaluate for all the observations. So, it will be summation of I equals to 1 to N Y I minus Y cap whole square. Because what we want is we want to reduce down this complete error for all the observations that we have, right? Now, when I'm saying what is this Y cap all about? It is nothing but summation of I equals to 1 to N Y I minus what is Y cap? MX plus C whole square. Can I say? It's complete square, right? Y I is nothing but Y cap is nothing but MX plus C. Now, what you can do you can just try to open this bracket. I hope everyone knows the formula of uh A minus B whole square. Do you know what is the formula for A minus B whole square? It is nothing but is equals to A square plus B square minus 2 AB, right? So, what we can do we can just open this formula. We will be able to get A square plus MX plus C whole square B square minus 2 into A into B MX plus C. Right? This is the complete value which we will be able to get. Now, if I'll ask you if I'll ask you This is the This is the formula This is the error that we will be able to get, right? This is the error or the residual that we will be able to get. This is the error. This is the error, right? Now, if I'll ask you, what is the derivative of this error with respect to m? What will you answer? What is the derivative of this m uh this particular error with respect to m? Now, I hope everyone knows how we will be able to calculate the derivative of any given value. For example, if I'll say, "Can you tell me what is the derivative of x squared with respect to x?" So, what you will say? The co- the uh you know, power value will come ahead. It will be 2 into x key power 2 minus 1. Subtraction by 1 will happen. So, it will be nothing but is equals to 2 into x. Similarly, derivative of any constant value is always equals to zero. Derivative of any constant value will always be equals to zero. This is very, very simple calculus which I think everyone is already aware of. Doing the same calculation, if I'll ask you, "What is the derivative of this e with respect to m?" Just try to pause this video and let me know what will be the answer you will be able to give to me. So, here if you will be able to see, this is a constant with respect to m because we are doing, it will be zero. Plus, it will go ahead 2 into mx plus c. 2 into mx plus c. Again, you have to do the derivative with respect to mx plus c, you will be able to get into the value of x. Into the value of x. Now, minus two y i x. Can I say? This is the value with respect to the derivative of 2 y i into mx plus c. With respect to mx, we will be able to get the answer as x only. C is constant, so what we will be able to get is 2 y i x. I hope it is clear to everyone. So, here if you will be able to observe if you will be able to observe, what we can do is we can take 2 into x as common. 2 into x i as common. What we will be able to get inside the bracket, what we will be able to get inside the bracket is we will be able to get c plus here we will be able to get mx right? minus y i. Can I say? Can I say? Yes or no? It is nothing but c plus mx minus y i. Can you see this this is the equation for what? mx plus c. This is the equation for y cap, right? So, can I say that the value which we will be able to get here is 2 into x i y cap minus y i. Can I say if this is the value which we will be able to get? This is the value which we will be able to get with respect to the derivative of error with respect to the value of m. Derivative of error with respect to the value of m. Yes or no? Right? Because mx plus c is the equation of y cap, right? Just now if you will remember, we have just now replaced this particular point. Similarly, if I'll ask from you, what is the derivative of error with respect to c? With respect to c, if I'll ask you, just try to evaluate this. Again, this is the constant. Achha, one more thing, make sure that we will be doing this for all the observations with the value of i equals to 1 to n. So, this is the value for all the observations that we have, right? So, here with respect to c again if you will be able to observe just focus on the equation i equals to 1 to n. What is happening? Again, for the constant it will be zero. Now, it will be 2 into mx plus c. Again, 2 into mx plus c. Plus again when you will do the derivative of mx plus c, you will be able to get the answer as one, right? Minus 2yi if I'm not wrong, right? 2yi will be the answer. So, this is the derivative with respect to c we will be able to get. Now, see if anyone will be able to uh you know, face any issue with respect to derivative part, it means your calculus is is not uh you know, revised. So, please make sure that you you will revise the concept of a calculus, then only you will be able to understand what I'm doing here. See, it's very very easy concept. Just remember how basically the differentiation usually happens, right? Wherever the constant is there, you have to make sure that it will be zero. Wherever the value of x is there with respect to whatever value you are doing, for example, with respect to m, you have to make sure accordingly you will do the do the derivative. With respect to c, accordingly you have to do the derivative, right? So, now here if you will be able to observe, if I'll just take two as common, what we will be able to get? mx plus c minus yi. Again, one amazing thing I'm seeing is I can write this equation as two times y cap minus yi. Right? This is the kind of a equation which we will be able to get with respect to the derivative of e with respect to c. So, what is happening? Are you able to understand? Will you be able to understand that wherever I have mentioned the value of derivative of e with respect to m, what you have to do? You can calculate that value with the help of this formula. With the help of this this formula. I hope now you know what is the value of xi, you know what is the value of y cap because you already aware about the value of m old and the value of c old. You know what is the value of yi, which is the difference between the Do you remember? The error is something the difference between the actual and the predicted again coming into picture. So, this is the This is how basically once you will insert all the values of xi, y cap, and yi, you will be able to get a derivative of e with respect to m. Similarly, you will be able to get the value of derivative of e with respect to c once you will fill the value of y cap and yi in this equation. And what you will be able to get is the new value of m and the new value of c, and accordingly, you will try to evaluate at each and every point of time what is the new error that you will be able to get via this formula. What is the formula to calculate the error? This is the error. So, what you will be able to observe? You will be able to observe that at every iteration, you are getting closer and closer and closer value of error approximately equal to zero. At any point of time, you will reach to the point where it is minima. That is something which you can you can take it as a global minima where the error will be minimum. That is what we want to achieve because that is where you will be able to get our final value of m and c. That is where we will be able to get the final best fit line. That is where we will be able to get a very good accuracy of our model. That is where our model will behave very well. And that is what we call it as a learning of a model. So, in future, if someone will ask you that, "Okay, whenever someone is saying that I'm training my linear regression model," automatically, your brain should trigger in a way that, "Okay, he or she is training in such a way that they will be able to reach to a most optimal value of m and c." Always remember this statement. Okay? Many people I'm saying again, many people are not aware about this fact. They just know how they can import the linear regression from sklearn library and how they can do the implementation, but this actual maths, not everyone is aware about this. So, please make sure that you will be easily able to understand now with this lecture that what I'm trying to convey here. Right? So, what we have understood so far in this session, I think it's a bit long session. I'll try to wrap up now, and upcoming things I'll try to share in the upcoming video. So, what things I have covered up in this session? If you will try to understand it again, we have talked about with a very simple example. We have seen that whenever we're talking about a linear regression algorithm, means we are trying to get the predictions of a continuous real number. Okay? Okay, we try to understand that point. After that, I have taken a very simple example of a height and weight. Let's say height is an input value and weight is something which I'm trying to predict. So, what we will do is we will try to have let's say thousands of records. We will try to train those records. Now, what is the meaning of actual training in a linear regression? Means we're trying to get that best fit line. How will we able to get the best fit line? Where our error will be minimum. And that's where a new word of error or residual I have introduced. The simple meaning is the gap between the actual and the predicted value is minimum. Now, we always have a actual value because we are dealing with a supervised machine learning algorithm. And we always have a predicted value where we will randomly initially assign the value of M and C to get a any line, okay? Which may not be a best fit line. Now, what will happen is with the help of gradient decent optimizer, what we are trying to do is we're trying to reduce down our error. How the error will get reduced? By updating the value of M and C. How the update will happen? It will pick up the older value of M minus eta times derivative of E with respect to M or derivative of E with respect to C. Now, here eta is something which is a learning rate. I'll talk about this factor of learning rate in a later part of the section of the video. But as of now, this is you can consider it as a constant value which is usually 0.001 in maximum scenarios. So, as of now, you can consider that this is a constant value which we have to multiply. We'll talk about this in a in a separate session that what is learning rate, how this value will get impacted to get the final answer? We will try to learn each and everything in a in a great detail about this particular part in a separate session. How this derivative of E with respect to M is evaluated? I have just now shown you the very simple example, very simple explanation of a mathematical part. Just we have to do is we have to first of all, uh, you know, completely broaden down the error value. After that, we will take, you know, very simply manner derivative of E with respect to M and derivative of E with respect to C. Now, in these two values, we will be able to get the value of the equation of MX + C - YI, which is Y cap - YI times of two in the with respect to C part times of two into XI with respect to the M part. Once you will, you know, fill the values of Y cap, YI, XI, and the value of two, at that number, we will be able to get that complete derivative of E with respect to M or derivative of E with respect to C. And that's how we will be able to get the newer version value of M and C. What we want is we want to to achieve that global minima, where the error will be minimum. So, with the help of this equation, we will be able to get that most optimal value of M and C, where the error will be minimum. So, I hope that now you will be able to get a clear-cut idea about what is linear regression, how will it work, and what is gradient descent, and how it is helping to get that error as a minimum value by updating the value of M and C. With this, let's end today's video. I hope that you find today's video insightful. If you do, please hit like button, and I'll try to come up with another video where we will try to, you know, talk about further concepts with respect to the same. Please make sure that you will watch all these videos videos in a sequence manner. If you will miss one video and come to the next video, you will face a trouble. So, that's my humble request to each and everyone. With this, bye-bye, everyone, and I'll see you all very soon in the upcoming videos.