How We’re Fooled By Statistics

Key Takeaways

[Music] which is most effective for helping people learn punishment or reward we consider the case of Israeli fighter pilot training because instructors there found that negative feedback was far superior to positive feedback if a Cadet performed a particularly poor maneuver they would reprimand him but they noticed that on follow-up attempts his performance invariably improved in contrast if they praised a Cadet for executing a skillful maneuver his performance on subsequent attempts typically declined so naturally They concluded that positive feedback is ineffective or even detrimental whereas negative feedback is what works the problem is this seems to contradict a body of research that shows positive feedback is actually more effective than negative feedback for example in studies involving teachers it was found that if a teacher increases the ratio of positive feedback to negative feedback that actually increases the percentage of time students spend on task in another study rugby players were given a video feedback session following a game except half of them were shown their highlights and they were praised for what they had done and the other half were shown their biggest blunders their mistakes and they were scolded for doing things incorrectly now a week later at the next game it was the players who had received the positive reinforcement who performed the best and in fact the difference was not just psychological blood tests revealed that the players who received positive feedback actually had higher levels of testosterone than players who had received the negative feedback I think that's quite remarkable that even a week post feedback you could still see a physiological difference between the two types of feedback wow well so why didn't this work for the Israeli fighter pilots well maybe there are cultural differences and In some cultures negative feedback works better or maybe it's task dependent perhaps some skills like learning to fly a fighter jet require more negative feedback or maybe the feedback had nothing to do with the performance of the fighter pilots at all maybe they would have performed in the same way regardless of the feedback they received go with me on this imagine you have a hundred students taking a test which consists of a hundred true or false questions on a subject they know nothing about now assuming they all answer randomly we know that the resulting distribution of scores will have an average of around 50 but just by chance some students will have scored significantly better or worse than the average man if you select the top 10 students whose scores were all above 50 and gave them a second very similar test you would find that the average of their scores would drop back to around 50 similarly if you selected the bottom 10 students their score on a subsequent test would rise to 50 this is just regression to the mean regression to the mean is the reason why if you have a really good round of golf today your round tomorrow will not be as good yes that is because random chance plays a role in virtually everything that we do so the outcome comes of events are influenced by both our skill and a little bit of luck so if you have especially good luck on one day chances are your level of luck will not be as good the next day now that sounds a little bit like the gamblers fallacy which states that past events influence future probabilities for example if you flip a coin and it comes up Tails a couple times the gambler's fallacy is to expect the probability of heads to increase for the next flip in reality it doesn't it's still 50/50 the idea with a Gam fallacy is that probabilities change so that overall things even out in the long run that's the idea with regression to the mean it's not that things are evening out it's just that extreme events are becoming diluted by the average events which happen much more commonly now with these students they were completely guessing on every question but even when you know something about the subject there's still going to be that element of chance and so regression to the mean always occurs just to a lesser extent this is really important to consider in research imagine you're triing a new drug to help prevent heart disease so from a sample of patients you select those in the bottom 10% of heart health indicators people with high blood pressure high cholesterol that sort of thing now after a month of being on the drug you test them again to find that their scores have improved well great the drug is working right well maybe not see the trouble is that although blood pressure and cholesterol are more stable than say your score on a random true or false test there's still some inherent variability caused by say your level of stress on the day or your recent diet or even the uncertainties introduced by the measuring apparatus so the people who ended up in the bottom 10% on the first test likely had these factors all count against them they were particularly unlucky but you should not expect them to be as unlucky when you test them a second time so their scores should improve just based on random chance that is why it is so important that clinical trials use control groups drawn from exactly the same population so that you can see whether the drug improv scor is more than random chance alone would or what about assessing the impact of speed cameras when they're first installed they're normally put in locations that have had a high volume of accidents in the previous year or two makes sense now those accidents are likely due at least in part to bad Road design but also due at least in part to bad luck and so you shouldn't be surprised that after a speed camera is installed the number of accidents goes down it would go down anyway simply due to regression to the mean meanwhile somewhere else that previously had a small number of accidents will likely have more so the overall accident rate may not change and yet we will feel as though our investment in Road Safety has paid off but perhaps what's most troubling about regression to the mean is how it influences our perception of feedback for example with the fighter pilots after they execute a maneuver particularly poorly chances are the next attempt is going to be better regardless of the feedback they receive similarly after an especially successful display chances are the next attempt is not going to be as good and that has nothing to do with the feedback and everything to do with the statistical nature of our universe but we are hardwired to see patterns and causality everywhere which is why the instructors felt that positive feedback was detrimental and negative feedback is what works and this is really unfortunate because that is the exact opposite of what well-controlled psychological studies show you know I think it's really unfortunate that if you give negative feedback chances are you'll be rewarded they will do better and if you give positive feedback chances are you'll be disappointed because it's difficult to maintain that level of luck but that's the way our world is so think about that next time before you tell someone off if you stay positive it may just work out for the best in the long run this video was inspired by the book Thinking Fast and Slow by Nobel prize winning Economist Daniel Conan so that was the first time I heard of the Israeli fighter pilot training and really started thinking about regression to the mean it was pretty awesome and I downloaded this book from audible so I was listening to it on my travels I'm currently at the perimeter Institute which you can see has some beautiful blackboards now if you want to listen to this book you can download it for free by going to audible.com/veritasium or you can listen to any other book of your choice for a one-month free trial so thanks to Audible for supporting me thanks to Perimeter Institute for putting me up in this amazing location and I will see you next time with some theoretical physics ideas so stay tuned for that