6. A/B Testing End to End Example Netflix Case Study

Welcome to video six. Let's take all the math we've learned, null hypotheses, P values, errors, and apply it to a real-world scenario. Let's pretend we are data scientists at Netflix. We have a hypothesis. Changing the thumbnail for Stranger Things will increase the play rate. We create two versions. Version A is our control, the current image. Version B is our variant, a new action shot. We can't just change it for everyone at once. That's risky. Instead, we run a controlled experiment. First, we need the randomization engine. As users visit the home page, we hash their user IDs to randomly assign them to either bucket A or bucket B. This randomization is critical. It ensures that the only difference between the two groups is the image they see. Everything else, time of day, device type, user taste, is averaged out by the law of large numbers. If we see a difference later, we know it was caused by our thumbnail, not bias. We let the experiment run for 7 days to capture full weekly cycles. As data flows in, we track the click-through rate, or CTR. Initially, the results are noisy, but as the sample size grows, our confidence intervals shrink. By day seven, we see that version B has a CTR of 12.8% compared to version A's 12.1%. That is a relative lift of nearly 6%, but is it real or just noise? Our P value calculation shows it is less than .05. The difference is statistically significant. Now comes the decision. First, we check statistical significance. P value is low, check. Second, we check practical significance. Is a 6% lift worth the engineering effort? Yes. Third, we check guardrail metrics. Did this change accidentally slow down the app or increase cancellations? No. All lights are green, we reject the null hypothesis and launch version B to 100% of users. In the next video, we will tackle the most popular test of all, the Student's t-test. I will see you there. If you found this video helpful, please like and subscribe to the channel for more AI engineering content. Thanks for watching.