The 4 main tasks in the production ML lifecycle

Key Takeaways

in in the chat so I'm interested in what you discovered in terms of I mean at a at a basic level of packing but pattern recognition just what what the main tasks that people do in the production machine learning life cycle are what did you what did you find out I think a good answer to this is what the textbook says and they found that it's different from what the textbook says and what the textbook will tell you for machine learning it's first you collect data step one step two is you train a model step three is you evaluate that model on a holdout data set make sure there's no overfitting and then step four is you deploy um and the what we found is that we can still categorize into four steps um and maybe the data collection part is similar except for it's more of a look like every I don't know a week or so we want to collect new data we want to make sure there's some QA on that data or something but the rest of the steps the last three steps are totally different um the Second Step what I said before model training is actually experimentation in general whether it be training new models whether it be trying to Source new data or adding new features there's a lot of ways you can think about improving a model um and a lot of the participants actually preferred to look into finding new data gave you signal or making features more fresh instead of stale features that they had before so that's kind of step two stage three in the process was we call evaluation and deployment so evaluation is not a one and done thing what happens is evaluation is kind of done maybe on a holdout data set at first and then it's deployed to a small fraction of users and then when the model shows a little promise there increasingly it's deployed to more and more users as we learn more about what it can do what it can't do what failure modes exist how do we go and Patch problems until we've kind of gotten to the full population um so key takeaway is evaluation is not a one-time thing it is a loop on evaluation and deployment a multi-stage deployment and then the latest the step four we found was this overall monitoring and response stage which was when you do have these models in production what is their what is their live performance um if you see the performance dropping what are the bugs where are the bugs how do we respond to them quickly whether that be actually trying to go do root cause analysis or simply retraining the model there is a stage around making sure that there's little downtime for these services so we do we have those four stages shown in the first figure in the paper and and it was interesting to several of us authors that they don't match the textbook I think that's like kind of a narrative that we want people to take away absolutely and let me ask are these um different steps I mean there's an iterative Loop happening there but are they kind of coupled in someone because you could imagine monitoring and validation yeah aren't always separable right totally um I'll say that monitoring is kind of done across the state or kind of in all of the stages um people monitoring their training jobs people monitoring data collection a lot of the times there's human in the loop processes to collect and label data to verify some quality whenever you see a failure um on the ground to go and collect examples that look like that failure so you can go back and augment your validation sets so in that sense there's monitoring all over the place but the one stage that we found was that was super iterative in itself was um evaluation and deployment um evaluations data sets never stay the same they're always changing they're always growing especially in tasks or domains where failures have such a high cost like autonomous vehicles are a great example of a failure is a really high cost when we observe one we need to make sure that we have no more failures like that so how do we go and invest efforts into making sure our foldout data sets whenever you evaluate your bottles new models in the future they're also robust to the problem yeah and I suppose drilling down even a bit more into data collection and I think this is one thing that that you're getting at in textbook data validation you're given a holdout set right as opposed to be actively getting that data and validating it and making sure it's the right data then using it as validation set there are feedback delays there all of these totally totally um I feel like I was never taught this in a machine learning class nobody taught me how to actually evaluate a model it's not like I need a big checklist um I just want to know if I were to be a machine learning engineer for a month not just one time how do I what do I do about my validation set do I keep it the same do I grow it when do I add to it what do I add to it I think this in itself super interesting problems for people to consider