Breaking Linear Regression

Key Takeaways

hello everyone in this video we're going to look at the assumptions of linear regression and see what happens if we break them at least its effects in the context of machine learning so here I have four assumptions that I'm breaking down with linear regression that's the linearity with parameters independent residuals normality of residuals and the equal variance of residuals so let's get to it now the goal in machine learning and regression specifically is to draw some curve that fits into a cloud of data points now with linear regression this is going to take a very specific form probably like one of the forms it can take is Theta 1 x i plus Theta zero it's just a linear equation right here both linear in terms of its parameters as well as the predictors and if we were to actually plot out this curve it might look something that looks like this just a straight line over here now another form that this potentially could take is of the form of theta 2 x i squared plus Theta 1 x i plus Theta 0. now this is an equation that is still linear in the parameters even though it is a quadratic in terms of the predictors and what you'll get if you plot this out it might be something that looks like a little curved line that looks like this now both of these are still acceptable forms for linear regression and do not violate the linearity Assumption of linearity with respect to parameters but one example that does violate the linearity with respect to parameters is let's say that if this was Theta 1 squared x i plus Theta 0. now this equation is now quadratic with respect to the parameters and it's not something that we can handle here now what happens actually if this case arises so let's talk about that so in a normal case of typical linear regression let's just assume that we took the simplest and linear form of regression that is linear in terms of both parameters and the predictor now the goal of linear regression is just to find the estimated parameters of theta 0 and Theta 1 which we typically find with maximum likelihood estimation and doing some math which I'll probably talk about later you'll see that it you can find it very easily just by minimizing the residual sum of squares so that's a sum of every single parameters actual value minus its predicted value and squaring it now in order to deal with Maximas and minimas we typically take derivatives with respect to each parameter now if you take the derivative with respect to Theta 0 you're going to get just some linear equation in terms of theta 0 and Theta 1. and if you also do the partial derivative with respect to Theta one you're also going to end up with another linear equation and when you have two linear equations and two unknowns it's really easy to solve with linear algebra and you can get closed form Solutions however in our case that we want to break linear regression we have now a Theta 1 square turn up here what will happen is now we need to do an argument of theta 1 square over here an argument of this entire expression and now if you try to take the partial derivatives you're going to say if you take it with respect to Theta zero you'll probably get a quadratic equation and if you take it with respect to Theta 1 the derivative with respect to Theta 1 you're going to get even a cubic equation and because you get non-linear terms with respect to the parameters that you want to solve you cannot solve this with simple linear algebra and there won't be a closed form solution that you would typically want to see in linear regression that said you can still technically put this data into a linear regressor and get some solution using some estimation techniques like gradient descent but basically this rule of linearity exists for the Simplicity of kind of determining the terms Theta 0 and Theta 1 with linear algebra now let's talk about the other three assumptions I put it as the i n and E assumptions but all of these are related to residuals so what are residuals so let's say that you have a data point that's up here and this line over here this green line is essentially going to be the prediction of what your linear regression would predict at a certain point of x and for this certain point of X you can see that this here is the difference between the predicted value and the actual value and it will be your residual which is Epsilon I now let's talk about each of these assumptions individually so first we'll start with I so the I here is independence and it basically says that these Theta these Epsilon terms are independent of each other and what this actually means in the context of looking at this graph is that well this here is a constant when the value of x is given and so it means that all the Y's over here are also independent of each other and so for a given X if the Y's are independent of each other that means all of the points on this one line all the data points that we would potentially see are all going to be independent of each other that's the first assumption next assumption is going to be our normality assumption and this basically says that this residual over here is going to be distributed normally with a mean of zero and some value of Sigma I squared which is going to be the variance and if you bubble this up this kind of leads to a fact that well y i given x i is essentially going to be sampled from also a normal distribution and if you add a constant because we're given x i so this entire term is a constant if you add it to a normal distribution it's technically going to be added to its mean so this plus 0 is which going to be Theta 1 x i plus 0 Theta 0 and this variance is going to remain the same essentially what this is saying visually is that all of the actual terms of Y are going to be normally distributed for given value of x so again along this straight line over here we can kind of write a little bell curve that kind of looks like this and for this given value of X1 we can also say that these points are going to be distributed like this the peak of the curve is going to be on the green line it's on the estimation because that's exactly what our mean is and so the mean should kind of coincide with the Peaks over here and Visually that would mean that there's like a lot of points that are concentrated here and a fewer points that are concentrated on the sides same for every single point of x a lot towards the center little of these y's towards the side so that's the second assumption now the third assumption is going to be equal variances so what this essentially means is that you see this I over here we're not going to write all this same over here we're not going to write this because the variance of well how that bubbles out visually is that the variance or the spread of this normal distribution is going to be the same as the spread of this normal distribution and is going to be the same as a spread of well any kind of normal distribution that you might add even if it was here for example so those are effectively the three assumptions that we need to follow during linear regression but like why do we need to solve this and what happens if we break this so the goal of linear regression like I mentioned before is going to be to find the parameters Theta 0 and Theta 1 which we find with a technique called a maximum likelihood estimation and here we want to maximize typically the likelihood of the curve fitting the data which is proportional to the probability of seeing the data itself and so it's the problem the product of the probability of seeing all the N data points and because P here is a probability density function that we've assumed to be normally distributed we get this equation over here which is the probability density function for a normal distribution and we want to maximize this or rather we want to find the values of theta 0 and Theta 1 that maximize this equation and what you can do is just take logarithms to convert this product into a sum term and you'll get a negative sign if you do logarithms so the max here just becomes Min and then we ultimately get the fact that we need to minimize the residual sum of squares in order to find the maximum likelihood estimates of theta 0 and Theta 1. and from here we we already know that this is a super popular equation that we always see when we're looking at linear regression so what happens if our assumptions now are broken well for one we already know this line assumption the L in line is linearity and we know that without the linearity Assumption we wouldn't be able to write this equation like we mentioned before because this would probably be a quadratic equation that might look like this and it just becomes more difficult to solve and you cannot solve it with linear algebra now we have this term I over here I is the independence assumption independence of residuals specifically but this bubbles up actually in this case such that we would not be able to write this form because here this probability should be the joint probability of seeing all of your data however you cannot represent a joint probability as a product of the individual probability distribution functions if they are not independent of each other and so without this Independence assumption you can't write this product and thus the computation just becomes so much more complex now moving on let's see what happens if we violate the normality assumption well we talked about before how this equation over here was the probability density function for a normal distribution and we wouldn't be able to now write this equation so explicitly and so cleanly because if there are multiple normal distributions going on or if one distribution is say some other uniform distribution another is normal then this equation also becomes far more complex and then finally we have this this e term over here which is equal variances and well with equal variances well we wouldn't be able to write this Sigma term on its own and we would need to write something like Sigma I same over here or rather Sigma n in this case and if you have a sigma n over here that means you can't really take it out of this product term notation and so the computation of this also becomes more expensive so I hope you can see now that in order to get this equation of residual sum of squares that we want to minimize the one that we know and love there are so many pieces that are required in order to actually get this more simplistic equation so I hope you can keep this in mind of like what are the assumptions now that linear regression is making and also why it is important that we need them for this specific derivation so thank you all so much for watching I'm gonna make some more videos on linear regression in the future with respect to math I love math so thank you for supporting and I will see you in the next one bye-bye foreign