The Main Ideas of Fitting a Line to Data (The Main Ideas of Least Squares and Linear Regression.)

Key Takeaways

when you go on a quest and that Quest is really awesome it's a stack Quest yeah yeah yeah hello and welcome to stack Quest stack Quest is brought to you by the friendly folks in the genetics department at the University of North Carolina at Chapel Hill today we're going to talk about fitting a line to data AKA least squares AKA a linear regression now let's get to it okay you worked really hard you did the experiment and now you got some data here it is plotted on an XY graph we usually like to add a line to our data so we can see what the trend is but is this the best line we should use or does this new line fit the data even better or what about this line is it better or worse than the other options a horizontal line that cuts through the average y value of our data is probably the worst fit of all however it gives us a good starting point for talking about how to find the optimal line to fit our data so now let's focus on this horizontal line it cuts through the average y value which is 3.5 let's just call this point B because different data sets will have different average values on the Y AIS that is to say the Y value for this line is B and for this particular data set b equal 3.5 we can measure how well this line fits the data by seeing how close it is to the data points we'll start with the point in the lower leftand corner of the graph with coordinates X1 y1 we can now draw a line from this point up to the line that cuts across the average yvalue for this data set the distance between the line and the first data point equals B minus y1 the distance between the line and the second data point is B minus Y2 so far the total distance between the data points and the line is the sum of the two distances and we can calculate the distance between the line and the third point that equals B minus Y3 now we've added the third distance to our total sum the distance for the fourth point is B minus y4 note y4 is greater than b because it's above the horizontal line so this value will be negative that's no good since it will subtract from the total and make the overall fit appear better than it really is the fifth data point is even higher relative to the horizontal line This distance is going to be very negative back in the day when they were first working this out they probably tried taking the absolute value of everything and then discovered that it made the math pretty tricky so they ended up squaring each term squaring ensures that each term is positive here's the equation that shows the total distance the data points have from the horizontal line in this specific example 24.6 2 is our measure of how well this line fits the data it's called the sum of squared residuals because the residuals are the differences between the real data and the line and we are summing the square of these values now let's see how good the fit is if we rotate the line a little bit in this case the sum of squared residuals equals 18.72% 05 that value keeps going down the more we rotate the line what if we rotate the line a whole lot well as you can see the fit gets worse in this case the sum of squared residuals is 31.7 one so there's a sweet spot in between horizontal and two vertical to find that sweet spot let's start with the generic line equation this is y = ax or a * x + B A is the slope of the line and B is the Y intercept of the line that's the location on the Y AIS that the line crosses when x equals 0 we want to find the optimal values for A and B so that we minimize the sum of squared residuals in more general math terms the sum of squared residuals is this complicated mathematical equation but it's actually not that complicated this first part is the value of the line at X1 and this second part is the observed value at X1 so really all we're doing in this part of the equation is calculating the distance between the line and the observed value so this is no big deal since we want the line that will give us the small smallest sum of squares this method for finding the best values for A and B is called least squares if we plotted the sum of squared residuals versus each rotation we'd get something like this where on the Y AIS we have the sum of squared residuals and on the x axis we've got each different rotation of the line we see that the sum of squared residuals goes down when we start rotating the line but that it's possible to rotate the line tooo far and the sum of squared residual starts going back up again how do we find the optimal rotation for the line well we take the derivative of this function the derivative tells us the slope of the function at every point the slope at the point on the far left side is pretty steep as we move to the right we see that the slope isn't as steep the slope at the best point where we have the least squares is zero after that the slope starts getting steep again let's go back to that middle point where we have the least squares value and the slope is zero remember the different rotations are just different values for a the slope and B The Intercept we can use a 3D graph to show how different values for the slope and intercept result in different sums of squares in this graph The Intercept is the z-axis so it's going back sort of deep into your computer screen and if we select one value for the intercept for example assume we set the intercept value to be three then we could change values for the slope and see how an intercept of three plus different values for the slope SL would affect the sum of squared residuals anyways we do that for Bunches of different intercepts and slopes taking the derivatives of both the slope and the intercepts tells us where the optimal values are for the best fit note no one ever solves this problem by hand this is done on a computer so for most people it's not essential to know how to take these derivatives however it's essential to understand the concepts big important concept number one we want to minimize the square of the distance between the observed values and the line big important concept number two we do this by taking the derivative and finding where it is equal to zero the final line minimizes the sums of squares it gives the least squares between it and the real data in this case the line is defined by the following equation y = 0.77 * x + 0.66 hooray we've made it to the end of another stat Quest tune in next time for another exciting adventure and statistics land