The 3 factors that Determine the success of ML projects

Key Takeaways

is you've identified three Key properties of ML workflows and infrastructure that dictate how successful deployments are yeah um what are they and and why are they so um we we have these three views they're not even related or we didn't even know about the Big Data 3vs um Joe was telling Rolando and me hey there's Big Data 3vs are you sure you want to make three v's and then we're like oh shoot we shouldn't make three of these but then on second thought we were like I think it's cool if we'd say that we came up with these three V's independently because it kind of highlights the synergies between machine learning and traditional data um but these three V's are Velocity validating early and versioning um and why did we come up with these three V's we wanted some way to explain kind of best practices and pinpoints among we were looking for patterns in what our interviewees said and since we asked such open-ended questions they're in the appendix but we asked open-ended questions like tell me about a bug you had last week something that caused you things like that are so open-ended they're so hard to extract patterns from um so it helped us to come up with these variables like velocity people kept mentioning they needed to iterate quickly on experiments because they had a large Frontier of ideas to try and they wanted to see something that would give a production lift so we wrote down like velocity velocity velocity when we were looking through our codes at the end of it they were like oh when people are doing experimentation they care about velocity and it really resonated up with us when we started thinking about ml Ops tools what makes an ml apps tool successful well experiment tracking is a nice space because it really 10xes your experimentation at velocity now I don't have to go copy paste into Google Sheets and back maybe that works if I'm the only person working on my model but at the moment that multiple people are working on an ml pipeline or model system then it's super nice to centralize all of the experimentation we do so we can share the knowledge that we've had so we had velocity for that for validating early um a lot of people complained about the fact that at their organization are there too many bad models made it to production or uh so that was like it was validating too late or um models were validated way too early and that they couldn't get anything to production so for one example in an autonomous vehicle company um the cost of deploying a bad model is so high um so they Incorporated all these checks they made evaluation take much longer they decreased the velocity and Engineers were grumpy but at the end of the day I think there's a quote in the paper that says that you know we'd much rather gate the velocity if it means that we don't get failures on the road um so again different tasks they they have different priorities where and I think that's also why people keep talking about how like machine learning like you know it's not even generalizable it's so different for different tasks um and when you think about it through the lens of these these it's not that if it makes total sense for different tasks they just have different priorities some people prioritize velocity over validation if it means if it's like a Rex's problem or something where the stakes aren't so bad if there's a failure um for sure so in that sense we really liked um this kind of framework of evaluating tools evaluating what people cared about and as people who like to build tools ourselves um there's some tool ideas that I've had that now I can confidently say that oh this is really not a 10x Improvement in people's workflows it doesn't really help their velocity doesn't validate better and it doesn't help people manage any more versions so why bother um and I really like that way of thinking about it