5. Statistical Errors & Power Analysis Type I vs Type II

Welcome to video five. Now that we understand sampling distributions, we need to talk about the decisions we make with them and more importantly the errors we make. This is the error matrix for hypothesis testing. It's exactly like a confusion matrix in machine learning. Across the top we have the truth. Either the null is true, no effect, or the alternative is true, real effect. On the side we have our decision. Ideally, we want to be in the green boxes, rejecting the null when there is an effect and doing nothing when there isn't. But probability is never 100%. We will make mistakes. The first mistake is the type one error denoted by alpha. This is the false positive. It happens when the null is actually true. Your new model is garbage, but random noise makes it look good and you decide to ship it. Why is this dangerous? because you just degraded the user experience. You added complexity to your codebase. You might be paying extra for GPU inference all for a model that does nothing. In medicine, this is approving a drug that doesn't work. We typically cap this risk at 5% which is where that famous 05 p value threshold comes from. The second mistake is the type two error denoted by beta. This is the false negative. This happens when your new model essentially is better. There is a diamond in the rough, but your test failed to detect it. Maybe your sample size was too small or the data was too noisy. This is a missed opportunity. You threw away a feature that could have made the company millions. While type one errors are active mistakes, breaking things, type two errors are passive mistakes, failing to improve. ML engineers often ignore beta. But in a competitive market, missing out on innovation is fatal. The inverse of type two error is called power. One minus beta. Power is arguably the most important metric for planning an experiment. It answers, if my model acts positively, what is the probability I will actually notice it? Visually, power is the blue shaded region. If you have low power, you are running experiments that stay inconclusive even when you are right. That is a waste of time. You generally want power to be at least 80%. If it's lower, you need to either get more data or look for a bigger effect. Here's the hard truth. For a fixed sample size, you cannot fix both errors perfectly. It's a trade-off. If you make your strictness alpha very low to avoid false alarms, you inevitably increase your false negatives beta. You play it so safe you never ship anything. The only way to cheat this trade-off is to increase your sample size n. More data shrinks the noise, allowing you to have both low alpha and high power. In the next video, we will apply all of this to a realworld Netflix AB testing scenario. See you there. If you found this video helpful, please like and subscribe to the channel for more AI engineering content.