P-Value Problems: Crash Course Statistics #22

Key Takeaways

Hi, I'm Adrienne Hill and welcome back to Crash Course Statistics. To recap from last time, P values tell us how rare something is. So far, we've been using that information to decide whether or not our hypotheses are reasonable and using P values to reject or fail to reject an idea. Today, we're going to explore P values a little more and talk about the logic of P values and some of the problems that come up. Remember to calculate a p value, we first assume that the null distribution is the true distribution our sample was taken from. Then we calculate how often we'd see a value that is at least as extreme as our observed value. So in probability terms, the p value is the probability of getting a sample as or more extreme than ours given that the null hypothesis is true. So all the values that we see in the sampling distribution are means we could actually get if the null hypothesis was true. For example, let's say the average cat weighs 10 lb or 4 1/2 kg. We might want to calculate the probability of getting a group of 30 randomly selected calico cats who have an average weight of 11 lb or 5 kg. If calico cats have the same average weight as the whole population of cats, the first issue is if in real life there's no connection between two things like fur color and weight, we still might get samples of calico, mackerel tabies, or tortoise shells that are different enough to cause us to reject the null hypothesis that there is no difference. Our alpha tells us how often this will happen. Let's say our hypothesis is that the reaction time of older professional chess players is different from the reaction time of the general population of professional chess players. Even if older chess players are the same as their colleagues, if we ran the study over and over and over, we'd expect that 5% of the time we'd mistakenly reject the null if it were true. This is one reason why p values are pretty controversial in the statistical community right now. Not everyone agrees that a p value less than 0.05 is sufficient evidence to reject the null hypothesis. In fact, some studies that look at incredibly important things like new medications have already decided that an alpha of 0.05 isn't low enough. They want p values lower than 01. So that if the null hypothesis is true, they'll only mistakenly reject it 1% of the time. Still others argue that 0.005 is the better cutoff. As you can see, the standard cutoff is arbitrary. Null hypothesis significance testing requires that we draw a line in the sand somewhere, but it's not clear where arguments have been made that we can have different p value cut offs or alphas depending on the situation and that scientists should be allowed to justify their reasons for picking a certain cutoff. But on the whole, many fields that regularly use p values have some sort of official cutoff that they use. The second related issue is that a p value tells you how extreme your data would be if you assume the null hypothesis is true. But when you really think about it, that's not what we want to know. We want to know whether the null is correct or at least probably correct. In other words, the probability of the null given that we've seen our data. A p value of 0.02 O2 in a study on cancer rates in mice tells you that if your new drug didn't work and there was no difference between the cancer rates of mice on and off the drug, then you'd only expect 2% of identically run studies to produce a difference in cancer rates that's as or more extreme than the one you just observed. But we can't use these p values alone to tell us about the probability of the null being true or false, even though it can be tempting to think we can. One common misinterpretation of a p value is that it can tell you the probability that the null hypothesis is true. For example, if a random sample of tuna has a 10% higher mercury content than a random sample of mahi mahi, it would be incorrect to say that a p value of 0.02 in this case means there's only a 2% chance that the null hypothesis is true. This is an especially tempting misinterpretation because it feels like it maybe should be true. But again, when we calculate our p value, we've already assumed for a moment that the null hypothesis is true and that any sample differences we see are actually just due to random sampling variation. If our p value for the chess study was 01, that means that we already assumed older chess players were the same as the general population of chess players. So 01 can't tell us much about the probability that older chess players are the same as their colleagues. That would be like saying, assuming that the grass is green, what's the probability that the grass is green? It just doesn't make much sense. Similarly, P values can't tell you the probability that you've made an error given that you rejected the null. Again, this is because P values don't tell you about the probability of the null being true or false. If you've rejected the null hypothesis, like that drinking orange juice is not associated with higher levels of cavities than drinking coffee, either you did so correctly because there really is a difference between cavities and OJ and coffee drinkers, or you did so mistakenly because there really is no discernable difference. But P values, since they assume the null is true, don't tell you how likely either of those options is. Ronald Fischer, one of the first proponents of null hypothesis significant testing, wrote that quote, "In general, tests of significance are based on hypothetical probabilities calculated from their null hypothesis. They do not generally lead to any probability statements about the real world, but to a rational and well-defined measure of reluctance to the acceptance of the hypothesis they test. In other words, getting a p value of 0.04 doesn't mean that there's a 4% chance the null hypothesis is true. The probability we want to know is the opposite conditional probability from what a p value gives you. We want to know the probability of the null hypothesis given we got this data. But that's not what we get from the p value. We get the probability of the data given the null. For example, we calculate probability of data given older chess players are the same as the population of chess players. But we wish we could calculate probability of older chess players are the same as the population of chess players given the data. And while all the same pieces are there, they're not the same. This is made even more clear when you realize the probability of being a child given that you're at Chuck-E-Cheese is not the same as the probability of being at Chuck-E-Cheese given that you're a child. This is one reason why P values are so perplexing. they don't give us the probability that we truly want. There are some statistical methods that will give you the probability of a hypothesis given the data and we're going to talk about those a little later. A third issue is that if you reject the null, you still don't have much information about the alternative. When the data is pretty improbable under the null hypothesis, we reject the null and accept the hypothesis that the data came from another distribution that is not the null distribution. We call this the alternative distribution. and the hypothesis that goes with it gets called the alternative hypothesis. If we reject the null that Mrs. Smith and Mr. Kennedy give the same amount of homework each week, then the alternative is that they don't give the same amount each week, but we don't know whether the difference is 30 minutes, 25 minutes, 45 minutes. Or for example, we might want to know whether people who were primed with the words elderly, Florida, and retired walked more slowly than the average person [music] who takes 10 minutes to go around our office building with a standard deviation of 1 minute. We think people primed with those words will. We take a sample of 50 people, prime them, and set them off. Their mean time is 10.5 minutes, which corresponds to a p value of 0.0036. [music] We already decided beforehand to make our alpha our predetermined cutoff 0005. So our p value which is less than 0.5 allows us to reject the null hypothesis. In this case that people primed with words about being old take a mean of 10 minutes to walk around the building. But what now? Well, we've rejected the null hypothesis that the prime subjects take a mean of 10 minutes. The alternative hypothesis is just that their mean isn't 10. Our p values can't tell us anything else. A fourth common issue for p values is more about how we interpret nonsignificant p values. If our p- value isn't lower than our predetermined cutoff, our alpha, we fail to reject the null hypothesis. Notice that we say fail to reject, not accept. Null hypothesis testing doesn't allow us to accept or provide evidence that the null is true. Instead, we've only failed to provide evidence that it's false. Consider this. Your best friend makes a statement, "There are no black swans in China." You think she's wrong. So, you go to China and you look at a bunch of swans and none of them are black. You may at a certain point decide that you've seen so many swans that if there were black swans in China, it's unlikely that you wouldn't have seen one yet. But you can't prove there are no black swans until you've seen every single swan. Just like you can't prove the null is true, that there's no relationship between two variables, you can only show that you didn't find any evidence. It's false. But the absence of evidence is not the evidence of absence. Failing to reject the null hypothesis doesn't mean that there isn't an effect or relationship. It just means we didn't get enough evidence to say there definitely is one. If we looked whether bees produce more honey when it's warm than when it's cold, we could look at some data and calculate a p value of 0.25. 25. Since we decided beforehand that our alpha would be 01, we fail to reject the null hypothesis that bees produce the same amount of honey in hot and cold seasons, but we can't conclude there is no difference or even that it's unlikely that there's a difference. We can only conclude that we didn't find any evidence of one. Since null hypothesis significance testing is often the first type of statistical inference that people learn, it can seem pretty limiting to know that you can't provide any good evidence for the null hypothesis being true. In some cases, the null hypothesis might be what you actually want to demonstrate. For example, say there are two groups. People who play a souped-up bells and whistles version of a cognitive training game and those who play a less fancy version of the game. If these two groups have the same amount of improvement in cognitive abilities, which is our null hypothesis, that's really interesting. It means that researchers could feel comfortable using whichever version of the game that they want. If playing the fancier, more aesthetically pleasing game made people with strokes or children with learning differences more likely to play it, researchers would know that that's fine. They wouldn't have any concerns that the bells and whistles would detract from the cognitive benefits. P values can be perplexing, but they give us insight into how to make decisions about data. They also remind us that people's perception of evidence can be arbitrary. What you consider sufficient evidence might not be enough to convince someone else. When you read about the results of scientific studies, you can see the alpha they used and decide if you think it's a stringent enough criteria. More than that, though, we now know what P values are and how to interpret them. This helps us compare the logic of null hypothesis significance testing with how we normally reason about the world. Thanks for watching. I'll see you next time. Crash Course Statistics is filmed in the Chad and Stacy Immigol studio in Indianapolis, [music] Indiana, and it's made with the help of all these nice people. Our animation team is Thought Cafe. If you'd like to keep Crash Course free for everyone forever, you can support the series at Patreon, a crowdfunding platform that allows you to support the content you love. Thank you to all our patrons for your continued support. Crash Course is a production of Complexly. If you like content [music] designed to get you thinking, check out some of our other channels at complexly.com. [music] Thanks for watching.