Spearman's rank correlation coefficient- Statistics

Key Takeaways

anyone my name is Krishna and welcome to my youtube channel so that study in this particular video we'll be discussing about Spearman's rank correlation coefficient my previous video and statistics playlist have already discussed about Pearson correlation coefficient and we are actually understood and the Pearson correlation coefficient is basically having this particular formula state that is covariance of X comma Y divided by variance of X we multiply the variance of Y okay and we have understood that if you focus on this particular diagram if we have a linear property of all the points of x and y okay suppose we have actually discussed and if we have two independent features x and y and suppose if x increases and y also increases if it increases in a linear pattern linear way right at that time we will be getting a correlation as one okay we can actually say that this p value will be actually 1 now you can see that all the points are linearly available over here with respect to your x axis and y axis usually we will be getting as 1 suppose it is inversely proportional like this kind of stuff right here you can see that it is inversely proportional so the my my value will be minus 1 suppose all the points are not in the linear position and if it is in the order like when the X is increasing Y is increasing you will be getting the value between 0 to 1 and suppose if it is in the decreasing matter so if it is in decreasing matter like if the X increases and Y decreases at that time you will be getting our value between minus 1 to 0 now this was with respect to Pearson correlation coefficient and remember guys if you still are looking for the video I already uploaded this particular video here in statistics playlist you can search for what is Pearson correlation coefficient Krishna okay now the most important thing and the most better option than Pearson correlation coefficient is something for the Spearman rank correlation coefficient now you can see in this particular figure right you can see where even my data of x and y are not linearly increasing you can see that it is having a nonlinear or structure or diagram you can see over here is nonlinear right so even though we know that yes when the X is increasing my Y is also increasing right in this particular case if I try to apply my peers and and I'll be getting a value of 0.8 but if I apply Spearman correlation I'm actually getting it as one now the main thing is that how Spearman correlation correlation is actually able to get the value as one now Spearman correlation if if I just show you the formula make sure guys you receive the KPD okay as wonderful content if you don't understand any minutes now first of all we'll try to understand the fall in love Pearson correlation function so this is my formula again conveyance of X comma Y divided by standard deviation of X comma R now if we see the difference of Spearman rank here also we have covariance of rank of X okay so here you can see that instead of covariance of X comma Y divided by standard deviation of X and standard deviation of Y what we are doing is that we are taking the rank of X and rank of Y okay so that is the formula where covariance of rank of X and rectify and then we have variance of rank of X and ran profile we will try to understand what exactly is this rack okay so everything is same only a minor difference here we are actually trying to find out the Pearson correlation okay Pearson correlation or rank of X and rank of my that is what will actually give us parent correlation okay so here again I am repeating this the difference between Spearman and Pearson correlation is that here and I'm actually trying to find out Pearson correlation of rank of X and rank of 1 now the question arises is that what is the rank of X and rat of Y so for that what we will do is that will I'll try if I go down which there is a wonderful explanation about this here you can see that in this example the raw data in the table below is used to calculate the correlation between the IQ of the person with the number of hours spent in front of TV per week okay now this is my next feature this is my wife feature the next feature I have IQ in the Y feature I have powers of TB for me now here you have values like hundred and six 786 286 200 150 99 28 now if I want to find out the correlation on this if we apply Spearman correlation is not Pearson Spearman so if if I was applying pure self what would happen I would just use this particular formula simple formula right of X comma Y divided by standard deviation of variance of X multiplied by million so I could have to use this particular formula I could have got my pure simple relation value okay to not be positive or negative based on this data points okay but in this particular case when you see if I want to apply the Spearman Spearman right I will try to find out the first of all the rank of X and rank of one what is this rank of X and rank of one I'll just show you and these are the steps what you have to do okay so firstly evaluate ID square to do this you have to use the following steps first of all guys we have to sort the data of the first column so X of Y I have sorted it over here again 86 97 99 like this it is in the sorted order create a new column X I and assign it the ranked values 1 2 3 till 10 and what does it disappear say now see yes this is my smallest number right 86 is my smallest number so I am banking it as 1 97 is my smallest number I am ranking attached to I'm just giving rank in the form of ascending order in the form of ascending order so this is basically my rank of X now similarly rank of Y what I'll do I sort Y you can sort all the Y elements with respect to this and here I will again be giving my rank okay and remember there's this rank that you see over here I have just used one variable that is X of 5 okay and with respect to that I have made a rank of X of I over here but this hours TV per week right this rank that you see this now if you see this particular xx value over here this may be the sixth highest element okay so the rank is given as 6 then I have 28 this is will be the 8th highest element so like this kind of ranks we will actually create ok remember we have just sorted X i we have assigned rank in ascending order in case of Y again suppose xx is the element this is the sixth highest element we are just providing the wrong ok so you can see the next order data column and create a fourth columns familiar sight lines to this and now I have assigned the ranks to the x and y now the next thing is that we create another column which is called as D of I T of I is nothing but rank of sorry difference between brand of rank of a sigh and rank of weii okay so I'm actually calculating again and let me write it properly I'm just trying to find out a difference of rank of X I and rank of Y I this difference I'm actually computing so here you can see the difference or 2 minus 6 is minus 4 3 minus 8 is minus 5 4 minus 7 is 3 minus 3 and all this battle is on it and then I'm actually squaring it for squaring it I'll be getting this particular value now in order to find out the Spearman Spearman correlation okay we have a formula over here which I already showed right this particular formula I'd already shown if you go on the top right oh this is the formula that is 1 minus 6 multiplied by Sigma di square divided by n multiplied by n square minus 1 okay now when I do this you'll be able to see this particular formula now I'll be getting my Rho value that is with respect to the rank of X and rank of Phi and be getting somewhere on minus 0.175 okay well with p value as this point zero six two seven one two ate it with the T distribution the value is close to zero shows that the correlation between the IQ and the outspent watching TV is very very low so this basically shows that going don't focus on this also as the on T distribution focus on this over here you have got a value which is pretty much near a to 0 right that basically means show shows that the relation the correlation between IQ and T be outspent is very very low if suppose this correlation was getting somewhere around 0.95 then I could say that the correlation would be very very high if if the value was somewhere somewhere like minus 0.9 5 then this we could say that if the X is increasing Y is decreasing this kind of correlation will be there for that right so by finding the Spearman correlation the main advantage is that even though your data is non linearly distributed if it is basically having a nonlinear shape like take this particular example where you can you can see that it is just not linear it is non linear right if it is increasing in this particular order it is decreasing in this particular order you can see that for this I'm having point nine two for this I am having - point nine and I am using and computing uses the same formula what I've actually shown you in the structure one more example with respect to the outlines also okay so if I go on the top over here you can see that Spearman correlation for this variable you can see that it is showing point eight four four Pearson correlation it is showing 0.67 I can definitely see that it is linearly increasing but they are some amount of outliers also over here right they are some amount of outliers also so using the same formula and doing this but you can see that with the help of Pearson correlation with just focuses on linear aspects right this is actually having the value as point six seven so more 3er understanding of the correlation between x and y is clearly done in Spearman correlation so most of the things like whenever you're using heat map whenever you are actually trying to find out correlation it uses this technique that is Pearman correlation and it actually helps us to find out the real correlation between XM one okay so yes this was a small video for us Spearman rank correlation always remember the rank is the most important one here also we are trying to find out the Pearson correlation but we are considering the covariance of rank of X from our rank of Y divided by standard deviation of the rank of X and standard deviation of rank of one okay now this basically indicates that and if I if I show you one more example over here I shall go on the top you can see that what is happening in this particular case a small increase in X a small increase suppose this is my X 1 and X 2 right I know X 2 will always be greater than X 1 in terms of rank okay how much it is increasing how much it is decreasing that we cannot know here we are having a nonlinear increase right so similarly we are y2 and y1 right here also we know that the rank of y 2 will be greater than Y bar so by this so by this we will be able to explain it pretty much easily and make sure that you have explained in specific way most of the correlation that we have which we do in the exploratory data analysis uses this pair membrane correlation coefficient so yes this was all about this particular video I hope you liked it please do subscribe the channel if you are not releases class here in the next video have a great day I thank you and bye bye