[MINI] ANOVA

Key Takeaways

[Music] Data Skeptic mini episodes provide high-level descriptions of key concepts related to data science and skepticism. Today's topic is analysis of variance. [Music] So Linda, today we're going to discuss a useful tool for hypothesis testing called ANOVA. A hypothesis test, of course, is a statistical method for evaluating if theory you have is supported by the data or not. For example, if you wanted to hypothesize if, I don't know, left-handed people are better looking than right-handed people, you'd measure the attractiveness of a random sample of people, separated by handedness, and you'd compare. So, a nova helps you compare whether or not two groups are different. Uh, well, not usually. I think it actually might work. But the reason most people turn to ANOVA is because they want to compare greater than two groups. So if you want to compare two groups, you might use something like a t test, which we talked about in a previous episode. But for three or more groups, analysis of variance is really tuned for that situation. So you want to give an example of three groups? Sure. So I guess we could go back to we were talking about your barista days recently. And maybe the metric that's interesting is how long people waited in line. On average, how long did people wait before they got to get you to make their drink? So, Kyle's talking about when I used to a long time ago work at Starbucks. I don't know there was an average. No one ever told me how long people waited, but they said vaguely remember you shouldn't wait more than 5 minutes. Well, it's pretty quick actually to get up to the cash register. Well, was that true at your store cuz it was a really good store or were there other stores that had a bad reputation that might have been slower? Or maybe they just had a busy reputation, right? I mean, I worked at two different stores. One store was very slow, so that was easy because there was usually only two people in line max. Mhm. And I worked at another store where there was always uh like 10 people in line at the store with the 10 people in line. Was it that they had more people coming in or that the they ordered more complicated drinks? What was the nature of the delay? They just called it a high volume store. High volume store. And were they still five minutes to get your drink in hand there? that one. I don't know. Again, no one ever gave me the feedback. I imagine there might have been a manager who was who or like a regional manager who looked across these stores. Maybe they oversaw, let's say, 10 stores in an area, something like that. And they might want to compare the performance of these stores. Yeah. I mean, managers, I assume their job is to tell them when they're underperforming and overperforming and where they rank in their region. Yeah. Exactly. So, analysis of variance could be a useful technique. So, we've talked about the average wait time, the mean at each place. The other thing we have to consider is the variance. So, you said your store, was it real? Did they actually get people their drinks in 5 minutes? Because that sounds awfully fast. The DC one, I used to work in Washington DC. I don't know cuz it was such a long line. But I would say for the most part, most likely cuz we were working very quickly behind the counter and that was just the level of service that they wanted us to keep. Did you have more colleagues, more staff members at the one that was high velocity? Yes. Ah, so maybe it balanced out. Ah, so now that's really interesting. If they want to maintain that 5 minutes, we can measure if they successfully do it or not. But anyway, yeah, the variance, it had to be under five minutes. What was the average, do you think? Was it 5 minutes or was it 4 minutes? It was probably close to five. And what was the variance like? The over under, if you will. I mean, it depends on what kind of drink. True. If someone just wants a cup of coffee, that's already ready. So, how long for that? like within a minute of them ordering. Oh, what's the most complicated drink to make? I would say a Frappuccino was the hardest one. Just had more steps, that's all. How long would the Frappuccino take? I mean, it depends how good you are. All five minutes, maybe. No, I mean, depends how many Frappuccinos you had in line, right? If you had like 10 fra If there were 10 people in line and all 10 of them ordered a Frappuccino, then they would all have to wait a long time, right? Yep. But if there were 10 people, only one ordered a Frappuccino, then they could probably have it really quick. So, let's say Starbucks simplified its business and all they had was the basic coffee. Walk in, give your money, get the coffee, get out. That's not what Starbucks is at all, but like imagine a coffee house that's like that. Okay. What would be the average turnaround time then? Because they're all getting the same product just in different quantities. I mean, I would probably guess a minute cuz the coffee should already be ready. Yeah. Do what would you think would happen to the variance? How would it change order to order? I mean, it wouldn't vary that much because it's a cup of coffee. The only variance is if they run out, then they have to brew more. Right. Right. So, we might say it's like average of a minute with like plus or minus 10 seconds standard deviation. Maybe that could be believable if the menu was quite simple like that. What if store A averaged 1 minute and store B averaged 1 minute and 3 seconds? What would you think about those two stores? Nah, doesn't matter, right? Yeah, it's probably equivalent, especially with the variance so small. What about 1 minute versus three minutes? Well, that one you have to ask why. Yeah. Yeah. And now what if you've got a whole bunch of stores that range between 1 and 3 minutes? How do you know if any are statistically significant in being over or underperformers? I don't know. Tell me. Aha. Analysis of variance. The basic premise here is that if the variance is very precise like within 1 second, then you can rely on comparing the means pretty easily. We ask our question about the means, but we do our analysis by looking at the variance. For example, if the variance is really really big, meaning that sometimes people wait in line for a really long variable of time. Some people have a quick turnaround, some people have a long turnaround, then any difference could easily be explained just due to the day-to-day fluctuation. Like what if your store had a surge of Frappuccino people come in? Obviously, that would skew your average a little bit slower for that day. The whole system has a certain variance. By that I mean that if you pick any random Starbucks visitor or coffee house visitor, they're going to wait a certain amount of time and you can study the variance of of that average weight. But if you break it down by stores, you can have the between store or between group variance, but you also have a within group variance. So at a specific store, each order could take a little bit longer or shorter. And maybe that's because of the complexity of the drink. What are some things that made your turnaround times faster or slower? Well, how many people were ahead of you and what kind of order did they order? Maybe the expertise of the barista. Yeah. I mean, some people are more experienced, some people aren't. Let's say we measured your store and we measured all the stores, compared them to their own averages. You're part of a really effective team and your average day-to-day wait time is very small. So, it's it's always 5 minutes plus or minus 10 seconds. Very precise operation you have running there. Okay. Okay. So the within store variance is very small. But what about if you compare between the stores? So you divide the within store your very precise number. Your five minutes. Uh but the variance on it we said is very small, right? Let's just say like 10 seconds. Okay. So you're dividing 10 seconds. Yeah. By a couple of minutes. Let's say four minutes is the average variance between stores. Oh, okay. So that'll give you a really small number and that would tell you that hm these stores actually differ by a lot because we have this really precise store over here and we know it's always 5 minutes very close to it and then there's some other store somewhere else that has this 10-minute average. That's pretty clear that that store is just slower. Right now the question is when does it become statistically significant? And that is where you would take this ratio and look it up in the ANOVA table and find out whether or not the result is statistically significant. So that ratio is our summary statistic that we can look up. Now if the ratio was one, that would mean we have equal within group and between group variance. Now if they aren't equal, one is either very big or very small. And that's how we can determine if we're going to reject the null hypothesis or not. So what do you mean to reject the null hypothesis in this situation? Yeah. Yeah. Good question. So the null hypothesis states that any observed difference between the the stores is just explainable by chance. That's like the case where your store takes five minutes, some other store takes five minutes, one second. It's probably small enough that it's not significant. The null hypothesis would say that any observed difference between stores is just due to chance. Luck of the draw. You could measure again a different day and see a different result. The data you have doesn't support a difference between stores. They all seem to work equally well. Now, there could be a difference, but maybe it's too small that your data doesn't let you to observe it. like maybe the stores differ by half a millisecond and you just can't you don't have enough observations to measure that. We never say we accept the null hypothesis. We just say we fail to reject it. On the other hand, if that ratio is really extreme in one direction or the other, then we do reject the null hypothesis, which means that we say that there is a statistically significant difference with between at least one of those stores and all of the group that that was compared. that one of them is either underperforming, you know, maybe they're slower or they're overperforming that perhaps a model store that the manager would love to know the the secrets of the operations there. So, let's say I oversee 10 Starbuckses. How is this helpful? If you have 10, you can't go to those stores very often, right? If you went to all of them once a week, you could only spend half a day at each of them. So, you don't necessarily know what's going on at your stores very well. But if you sent the mystery shoppers like we talked about or you had some other way of getting measurements, you could measure the weight times, right? Mhm. Now, if you knew some of your stores were especially slow, I presume you'd want to speed them up, right? Mhm. Did you ever have a manager come in with some brilliant idea of how to speed up a store? Brilliant idea? No. Okay. Did you ever have a manager? Any idea? They basically just make you work faster. I.e. you need to master whatever skill you're in charge of. But I mean, how successful is work faster as a strategy? Because it seems like you could also make more mistakes. Well, they're going to fire you. So, if you make more mistakes, they're going to find someone else to replace you. Oh, so it's very cutthroat then, huh? Yeah, there's someone else. Oh, I didn't know. You don't want to You don't want to work fast. They're going to put someone else in your shoes. Yeah. They almost didn't hire me. Remember? Oh, yeah. Uh, tell the story of why you got in. Uh, well, I worked in North Carolina at a Starbucks that was not competitive and they hired me right away. Trained me. I wanted a summer job in DC and I tried to apply. So then I went up to Starbucks and I was like, "Hey, do you guys have any job openings?" And they were like, "No." And then I was like, "Wait, wait, wait, but I worked at another Starbucks and I'm already trained." And they're like, "Oh, okay. You could start tomorrow." Well, that's a lucky end then. So yeah, another way your manager could use analysis of variance is maybe you as a new employee, he wants to know if you're good, right? like maybe you need more training or or maybe as you say he he'd want to fire you or maybe you want it was the she by the way I want to correct you all right or maybe uh you should be employee of the month right they want might want to determine that sort of thing so that's what analysis of variance is good for and the key sort of intuition to walk away with is that it essentially comes down to getting the summary statistic of the ratio of the within and between group variances and the the intuition there is that if you have very precise people then the within group variance is small and would probably dominate the ratio. If you have a process that has a lot more variability, then the variance should tell the story about how uh individual groups perform to between groups. The summary statistic will help us determine if we can reject the null hypothesis or not. Meaning we can conclude that one or more of the individuals is performing notably different. So analysis of variance never came up in your performance reviews or anything. Nope. I think there's a cost to gathering this data. Ah, and that is a really interesting point. What would you estimate would be the cost to doing something like that. Do you know? I have no idea. I don't even know how someone would start measuring this. Well, there's the mystery shopper, right? You have to pay them. Well, you just have someone standing there and recording when everyone finishes their drinks. Yeah, maybe if you're lucky, you could build it into the register, but that has some bias, right? Some people close out orders before they're done. I bet. And there is Goodart's law at work, too. If you're going to measure employees based on how fast they can complete a transaction, then they're probably going to mark it complete before it actually is complete, right? Yep. As we talked about GoodArt's law recently. Anyway, I think that's a pretty good coverage of analysis of variance. As always, thank you for joining me, Lindy. Thank you. And until next time, I want to remind everyone to keep thinking skeptically of and with data. For more on this episode, visit dataskeepic.com. If you enjoyed the show, please give us a review on iTunes or Stitcher.