ROC Curves and Area Under the Curve (AUC) Explained

Key Takeaways

this video should help you to gain an intuitive understanding of rooc Curves and area under the curve also known as Au an Roc curve is a commonly used way to visualize the performance of a binary classifier meaning a classifier with two possible output classes for example let's pretend you built a classifier to predict whether a research paper will be admitted to a journal based on a variety of factors the features might be the length of the paper the number of authors the number of papers those authors have previously submitted to the journal Etc the response or output variable would be whether or not the paper was admitted let's first take a look at the bottom portion of this diagram and ignore everything except the blue and red distributions we'll pretend that every blue and red pixel represents a paper for which you want to predict the admission status this is your validation or hold out set so you know the true admission status of each paper the 250 red pixels are the papers that were actually admitted and the 250 blue pixels are the papers that were not admitted since this is your validation set you want to judge how well your model is doing by comparing your model's predictions to the true admission statuses of those 500 papers we'll assume that used the classification method such as logistic regression that can not only make a prediction for each paper but can also output a predicted probability of admission for each paper these blue and red distributions are one way to visualize how those predicted probabilities compare to the true statuses let's examine this plot in detail the xaxis represents your predicted probabilities and the Y AIS represents a count of observations kind of like a histogram let's estimate that the height at 0.1 is 10 pixels this plot tells you that there were 10 papers for which you predicted an admission probability of 0.1 and the true status for all 10 papers was negative meaning not admitted there were about 50 papers for which you predicted an admittance probability of 0.3 and none of those 50 were admitted there were about 20 papers for which you predicted a probability of 0.5 and half of those were admitted and the other half were not there were 50 papers for which you predicted a probability of 0.7 and all of those were admitted and so on based on this plot you might say that your classifier is doing quite well since it did a good job of separating the classes to actually make your class predictions you might set your threat Threshold at 0.5 and classify everything above 0.5 as admitted and everything below 0.5 is not admitted which is what most classification methods will do by default with that threshold your accuracy rate would be above 90% which is probably very good now let let's pretend that your classifier didn't do nearly as well and move the blue distribution you can see that there is a lot more overlap here and regardless of where you set your threshold your classification accuracy will be much lower than before now let's talk about the rooc curve that you see here in the upper left so what is an RC curve it is a a plot of the true positive rate on the y- AIS versus the false positive rate on the xais for every possible classification threshold as a reminder the true positive rate answers the question when the actual classification is positive meaning admitted how often does the classifier predict positive the false positive rate answers the question when the actual classification is negative meaning not admitted how often does the classifier incorrectly predict positive both the true positive rate and the false positive rate range from 0 to 1 to see how the ROC curve is actually generated let's set some example thresholds for classifying a paper as admitted a threshold of 0.8 would classify 50 papers as admitted and 450 papers as not admitted the true positive rate would be the red pixels to the right of the line divided by all red pixels or 50 / 250 which is 0.2 the false positive rate would be the blue pixels to the right of the line divided by by all blue pixels or 0 / 250 which is 0 thus we would plot a point at zero on the x-axis and 0.2 on the Y AIS which is right here let's set a different threshold of 0.5 that would classify 360 papers as admitted and 40 papers as not admitted the true positive rate would be 235 ided 250 or .94 the false positive rate would be 125 divided 250 or 0.5 thus we would plot a point at 0.5 on the xaxis and 0.94 on the Y AIS which is right here we've plotted two points but to generate the entire rooc curve all we have to do is plot the true positive rate versus the false positive rate for all possible classification thresholds which range from zero to one that is a huge benefit of using us an Roc curve to evaluate a classifier instead of a simpler metric such as misclassification rate in that an R curve visualizes all possible classification thresholds whereas misclassification rate only represents your error rate for a single threshold note that you can't actually see the thresholds used to generate the ROC curve anywhere on the curve itself now let's move the blue distribution back to where it was before because the classifier is doing a very good job of separating the blues and the Reds I can set a threshold of 0.6 have a true positive rate of 0.8 and still have a false positive rate of zero therefore a classifier that does a very good job separating the classes will have an rooc curve that hugs the upper left corner of the plot conversely a classifier that does a very poor job of separating the classes will have an Roc curve that is close to this black diagonal line that line essentially represents a classifier that does no better than random guessing naturally you might want to use the ROC curve to quantify the performance of a classifier and give a higher score for this classifier than this classifier that is the purpose of au which stands for area under the curve Au is literally just the percentage of this box that is under this curve this classifier has an Au of around 8 a very poor classifier has an au of around 0.5 and this classifier has an au of close to one there are two things I want to mention about this diagram first this diagram shows a case where your classes are perfectly balanced which is why the size of the blue and the red distributions are identic in most real world problems that is not the case for example if only 10% of papers were admitted the blue distribution would be nine times larger than the red distribution however that doesn't change how the ROC curve is generated a second note about this diagram is that it shows a case where your predicted probabilities have a very smooth shape similar to a normal distribution that was just for demonstration purposes the probabilities output by your classifier will not necessarily follow any particular shape to close I want to add three other important notes the first note is that the ROC curve and Au are insensitive to whether your predicted probabilities are properly calibrated to actually represent probabilities of class membership in other words the ROC curve and the Au would be identical even if your predicted probabilities ranged from 0.9 to 1 instead of 0 to one as long as the ordering of observations by predicted probability remain the same all the AOC metric cares about is how well your classifier separated the two classes and thus it is said to only be sensitive to rank ordering you can think of au as representing the probability that a classifier will rank a randomly chosen positive observation higher than a randomly chosen negative observation and thus it is a useful metric even for data sets with highly unbalanced classes the second note is that Roc curves can be extended to classification problems with three or more classes using what is called a one versus all approach that means if you have three classes you would create three Roc Curves in the first curve you would choose the first class as the positive class and group the other two classes together as the negative class in the second curve you would choose the second class as the positive class and group the other two classes together as the negative class and so on finally you might be wondering how you should set your classification threshold once you're ready to use it to predict out of sample data that's actually more of a business decision in that you have to decide whether you would rather minimize your false positive rate or maximize your true positive rate in our journal example it's not obvious what you should do but let's say your classifier was being used to predict whether a credit card transaction might be fraudulent and thus should be reviewed by the credit card holder the business decision might be to set the threshold very low that will result in a lot of false positives but that might be considered acceptable because it would maximize the true positive rate and thus minimize the number of cases in which a real instance of fraud was not flagged for review in the end you always have to choose a classification threshold but the ROC Curve will help you to visually understand the impact of that choice thanks very much to naon for creating this excellent visualization below this video I've link to it as well as a very readable paper that provides a much more in-depth treatment of Roc curves I also welcome your questions in the comments