The Future of ML Ops: Open Challenges and Opportunities

Key Takeaways

foreign webinar uh it will be about the future of ml Ops open challenges and opportunities because we've had we have entry who's the creator of dstack and was previously at chat Rings which is an amazing after Two's Company now we've got it was also Charles Wright who worked for waiting biases which I guess is one of the foremost companies in the mL of space and is now at full stack the brain which is an amazing course for you know interested in deep learning beyond the let's answer the basics but beyond the departure like do research they throw away your models and stuff like that and they've got Shreya who's a PhD student at Berkeley and who has just published a really great paper phone ml apps called Opera operationalizing machine learning an interview study so text everyone for coming we will start with some well pre-plane questions but if you have any program questions please send in on the Q a and you'll be sure to go to as many as possible uh so yeah hi folks uh I'm going to start with a very general and Broad question for the people who you know have joined but I'm not sure like what's ml Ops so yeah the first question will be what is analogs uh let's go alphabetically so Android first right yeah um hi everyone uh and CR first of all thanks for having um having us and great great opportunity to talk about mlops um people have to do in my lops and sometimes people don't have time to really reflect on what whatever made up says um so it's really good to actually have a conversation about that so right um a lot of people talking about mlops and sometimes um I think especially big beginners get confused about what mlops is and how to get started and also thanks thanks for for this um yeah um important question so if you ask me um I would probably give an answer that would probably resonate with what well you can read on the internet um here and there uh luckily there is a lot of resources um mlops has um a number of practices of best practices or um making machine learning easier to reproduce and easier to take into production that's having said that um it's it's a very let's say um yeah so it's it's it's very generic answer perhaps but but in in the end I think indeed it it is all about all of that practices that you can do in order to make it easier to get your model into production and of course that's a lot about tools that help you follow that practices but I think we will have a chance to talk about that so that that would be my age first answer so trout great yeah thanks Andre for taking the first crack at that question I did not expect to be answering questions this difficult this early in the morning for me um so I think it's something it's a term that was formed by analogy with devops so I think it's probably smart to give a definition by analogy with devops so in my view devops is what allows developers to actually own stuff that's happening in production there's like a bad old world in the scarcely remembered past in which people would like package up software and they would go to an I.T team to end up deployed uh to end up in user's hands and thanks to uh the work that's been done on on tooling and infrastructure that's no longer the world that most software lives in but a lot of ml does still live in that world where there's an ml research team or a data science team that makes something and then they hand it off to an engineering team they throw it over a wall and start working on something else and so I view ml Ops is trying to allow ml uh ml researchers or data scientists or whoever is doing model development in an organization to actually own the the full model in production and everything around it in Shreya I guess it's my turn to give the definition I think it's very similar to Charles at least my thoughts I think as any practices that allow ml pipelines to be sustained over long periods of time be it technical tooling operational organizational um that's why I think it's so broad you have such a long tail of applications a long tale of stages in the life cycle um yeah how do we do it super cool question thanks everyone and I guess the next question is when should one use or at least start caring about ml apps like what size of team of project on earning the research or more if like only if you care about you know predictionizing it and for this one we'll do reverse alphabetical so we start to show you so for a while I thought this had to do with organizational team size then we did our interview study paper and it doesn't have to do with this because we'll have small companies that have like pretty robust practices that allow data scientists to collaborate with each other to harden artifacts deploy them Etc um so I really just think it's whenever you as a company have a service that you want people to use for long periods of time or it's it's not just like a small pilot or a proof of concept or a dashboard that you show one time to someone um then then like envelopes is absolutely necessary but in practice right that if you have such like a hard boundary between no mlops and ml Ops right you're going to find it's super challenging to get something productionized or even get any system in production uh so so I don't know the right answer to like when do we start incorporating hard into engineering practices into our day-to-day workflow as we trade models but I do think it's like absolutely necessary to have any like Production service yeah I think um yeah it's substantially I agree with that that maybe historically it's been on the basis of like team size and organizational maturity but that's I think uh like an accident of History uh I think when it comes to being able to allow the person who is building models to be able to create something that users can interact with or stakeholders can interact with and give feedback and then incorporate that into the like iteration cycle for models I think that's something that needs to be done like as soon as possible the more tooling we can get to make that process as like easy as it can be the way that libraries like hugging face and pytorch lightning have made training easier and easier um uh like I think I think the better when it comes to the what uh I think Shreya called hardened engineering practices uh those those kinds of things maybe can can wait for when you have like larger numbers of models or coordination across like larger groups um those you don't necessarily need a uh a perfectly instrumented Pipeline with uh DVC models that correspond precisely to uh to containers in your AWS container registry and all that yeah right um so well I was listening to to Charles and Trey I was um um thinking also about like for example if we compare with our traditional development software development for example uh they're also also maybe similar concept like for example again maybe not exactly similar but still Agile development right um and the the when I a long time ago when I got introduced to that concept I was fascinated by that and and of course I was well I there was a lot of confusion okay so what is that when I should use it how I should exactly use that right um and because everybody was everybody agreed about that it should be agile but but well there were different ideologies like how exactly to have it right uh and nobody could agree um I mean some people could agree to certain extent but there was no a single right answer to that question that is why often typically when you're talking about methodology you're not talking about very exact way of doing things uh but rather of relying on different set of on a set of different practices which you decide yourself when to use one practice and not use some other so for example if again if we take that analogy with software development for example um there we we might have refactorings or using git or using tests um and and then you can for example ask yourself okay so when when should I have tests um and and then of course you should or well typically there's common sense and some rationality behind that when it makes sense to have tests so there's some empirical data to to back it up but but mostly it's common sense and common sense says that um you need to have tests that's that's what you have to agree with um and then as if you plan yourself to be a software developer in the future you better learn how to write tests and there is no there is no other way here uh um we we can say same about versioning data right for um sorry let's jump into mlops um versioning versioning code um so basically what I'm just trying to say is um we we have to understand these practices where they come from and um and then follow common sense about this particular situation but in in the let's say in a as a generalizer as a general rule it means that all of these practices are important and at certain point they have to be taken care of and that's why for example if we are talking about um raising the quality of software for example right where we use this methodology as a as a tool to to raise that that level of quality uh which means we we have to adopt this this tools and these practices so which means that we have to adopt their myops practices if we want to increase the level of quality uh in machine learning yeah would be my answer okay thanks everyone so I guess for my next question will be how can envelope students be used to improve publishing any substance the let's say scientificity like the the science likeness of the of machine learning in general and we'll go back to alphabetical um sorry Eric uh can you can we have this question uh maybe repeated um to make sure yeah so uh so how could mlabs help more on the let's say science site more help with publishing if guaranteeing um reproducibility and helping make machine learning substance a more scientific views yeah right so alphabets quarter means me uh I'm starting right um gets um again uh probably I'm not the best to answer that question if for example what we mean with the question is like uh Academia like from there from their Academia research point over here um if we can leverage envelopes to increase the quality of the research and reproducibility of it so it it's hard for me to to answer that because well I come mostly from their kind of production side of things um uh but but if you press me on that um I would say that mlops has rather nothing to do with research um I mean in general it's it's helpful but I I again I I can be totally wrong here but to me it seems that they when we're talking about the operations we're talking about the production so basically we're talking about something that needs to be taken into production so basically something as a part of the production cycle but but I would love to to hear other opinions on that yeah I think um mostly I would agree there are there are forms of like reproducibility of research that are less about the standards in like software Ops and more about just the standards in other research fields around reporting of what you have done um in such a way that other people can reproduce it that machine learning because of its Embrace of conferences and archive and Twitter as the way that research gets shared has kind of dropped a lot of those standards uh to uh dropping those standards maybe pretty much exclusively to the fields detriment the rapid pace of iteration has its own other benefits um but I think maybe the more direct place where I see an interaction between mlops and research is that mlaps is changing the way that people think about and build machine learning power technology um orienting more towards an iterative style orienting more towards uh maybe some form of continual learning uh and that should adjust the research agendas of people who are building uh building models perhaps if you want to be the the employees uh the one of the people who spearheaded designer imagenet that should maybe be building something that looks a lot more like a stream and a database for people to to learn off of so that's where I would see maybe more uh a more fruitful place for mlopson Academy to come together rather than um uh nips mandating that everybody incorporate uh weights and biases into their papers um the short answer is that I that there's a reason that I'm in like the databases field I think that like the nearest itml at all made tracks um aren't quite I don't even know if it is their prerogative or the most important thing for them to be thinking about these kinds of issues but jeffishment Fields right like from I don't know 1960s from God paper have been thinking about how do we make it easy how do we extract away all the internals of like storage execution how do we provide query engines such that users can interact with their data and I totally think ml fits very squarely into this Paradigm um for that yeah so for that reason I really think that there is a home in Academia to do research on not not just MLS best practices but um what kinds of data management systems can we build to help with these challenges um and there's a lot of the vlb music lab papers in the last two to three years on these kind of tools and systems there's new tracks like the scalable data science track and vldb uh you're specifically for these kinds of systems for large-scale data science or I don't even know a scale of data science um I don't know if this answer your question though I feel like it's probably a separate research I have a lot of mixed feelings on this I have a project that I am working on I haven't touched it in six months specifically what Charles was saying about like the streaming version of imagenet um I came in grad school like oh yeah I should we should definitely work on these things to incentivize ml researchers to work on the quote-unquote right problems um but I think that's a little bit misguided like who's to say what are the best problems are to work on like the whole point of research is inherently exploratory um people should do what their personal tastes are um and they're so I can't even reason about there's one correct way to do a continual machine learning like I can totally see a world in which we do match retraining and batch prediction and just sort of like materialize predictions then we just do this every hour compared to like on demand execution or On Demand free training right like there's there's no right answer that I have to these questions and I think that's what makes it hard to prescribe like here is a paradigm that uml researchers should adhere to here's how much work you should do but there's some people providing all sorts of benchmarks like the clear Benchmark uh I think nerves 2021 um is a good example of this I think there's also like different paradigms of how these are evaluated like might have been just a good example of like like different like evaluation um and then of course I think like a few people in my lab or broader UC Berkeley youth labs are working on the notion of like agenda or sorry evaluation um my friend Deborah she does a lot of work in this area so so I think it's definitely covered uh whether it makes sense for like ml Ops Focus solely on this I don't think so again it's like it's academia's personal taste like research is all just personal taste right so sorry I don't have an amazing things around here um so let's go now through the question of to be uh we will Begin by like what we have and so if everyone couldn't mention true or treaty I'm about to use they believe are very interesting important and why let's read it tools for the current tools or open problems that tools are trying to solve I would use current tools and then the next question will be about two windows great um okay so I think things that have been catching popularly popularity recently experiment tracking tools last two years or so like it really 10x velocity for any model developer um you don't have to go and copy your results to a Google spreadsheet right now you can simply instrument your like training script and then see kind of what model to choose and inherently experimental area um another one that I think is very interesting um is these like end-to-end kind of machine learning or envelopes kind of Frameworks keyfx is a good example of this um just the concept of having this at the end allows you to get forms of products forms of logging um that a very customized stack would not be able to easily get especially like if you allow your data scientist to use whatever tool they want whatever programming language for model development framework reality is right like your infrastructure team needs to support kind of the ability to get prominence for any single prediction through that highly customized stack so that's that's hard I think and tools are just kind of mitigating this problem whether or not they actually help people with ML Ops they do help people with some like necessary uh blogging stuff so those are two tools yeah I definitely agree that um experiment management is something that has been like solved well and there's been a ton of touring progress in the last couple years uh and I'm not just saying that because I worked at weights and biases um I think that there's been a lot of work with um with other tools with ML flow with comment with Neptune for really like uh yeah uh Estrella said 10xing attacking velocity I had a really disgusting system of interlinked Jupiter notebooks and uh and some and Google doc not even a spreadsheet just type in that day what happened take some screenshots it was rough it was not good um so that's that's I think one of the biggest wins and that sort of has proceeded we were kind of proceeding out from training like the Duels for writing neural networks not good five six years ago with uh pytorch and tensorflow 2. um the tools for what kind of engineer on top of that with Keras lightning hugging face that kind of layer uh simplified in the last couple years experiment management solidified in the last couple years it's kind of like in the like diagram everybody draws of the machine learning pipeline we're kind of moving down the pipeline um the place right like this where where I'm starting to see like some other good tooling is at the other end starting from like actual like production and uh and deployment of models I think uh for example the thing I'm I'm kind of most excited about recent developments in is like serverless setups for GPU accelerated workloads uh It just fits so neatly with the way you would like your machine learning models to run their functions that you want to run in the cloud so we should be using Cloud functions it seems like to me uh but we've been held back by the lack of tooling like all the way down to the level of effectively like hardware and the layers just above Hardware around like virtualization of gpus and starting to see that tooling percolate up from stuff built by Nvidia up to the stuff built by fast moving uh startups on top of that um maybe Triton kind of being the interface layer there the track and inference server I think the opportunity to substantially cut costs and make the early phases easier with um with better sort of inference tooling is the other side that I've seen some good progress on that um yeah for me it's quite difficult to answer this this question um because it's um again when we we it's really hard to to tell which Bristol is more important for most available uh for example if you go to the field and then look at how people actually do that job you'll realize that most of them are not using these fancy tools um at all and um people often used to think that sometimes they think they shouldn't never use but they have an alternative and then they have to to use them so for example SSH right um it's quite uh economics feeling about that technology right what it allows you to do is for example go to a remote machine and then basically use that resource of that remote machine to run what you have to move into and for example this is often used um basically the development experimentation when you're on different kind of scripts uh ad hoc or training at all and um it's not designed for and when whatever it helps for that matter yes at all um or sometimes people use things or or screen as a something to keep the session up you know and uh if you come to think this this is horrible to use these tools but uh people do use them uh and there's no there's not so many ways of basically not using them and basically that's what they use most of the day um they basically just work with let's say directly the wrong python script they watch this um well I mean um progress is just very slow and I often talking to some people and then like I'm I'm saying why are you using Studio to to to save your experimental networks of um uh like why I'm not using for example wooden voices and like what it and and I mean it's it's it sounds uh ridiculous but in fact most of the folks in the field actually is very old school old school very yeah I would I would rather call them old-school Technologies um so basically the the this is something that fascinates me that uh sometimes people ask okay so what's next in the machine learning I think it is just getting started and then it's basically we are about to see the well what is it basically their their race the race of their developer Tools in machine learning so basically what we see now I would rather say again this is very personal very subjective speculation but I would rather say that we are ahead of sorry not bad but we um may expect to see a lot of tools or or two other uh to how people do machine learning so basically um and basically I want to start to say that there is always uh two worlds for example at least two worlds their world how people do that most of the time how the majority of people do that so that that's very old way of doing things and there's always this modern you know kind of um um contage uh way of doing things basically people try very strange things some some of them actually end up becoming the mainstream tools but um so I'm very much interested to see which tools will go to the mainstream um so and for sure we'll wait and buy this and things like experiment tracking is is certainly one of them that already announced or started to go mainstream then of course um right now it's people are confused about deploying the model so what is there what is the easiest way of deploying my model and to be honest this is ridiculously um there's a ridiculous lack of answer to that question right now so basically yeah there are there are lots of solutions but there's no there's no proven ways of doing that now and I'm very much looking forward to see more more more standards more tools and services to of course to how people get the moral deployed I'll go for the next let's say prepared question before then going to the audience questions uh and Andrea has already touching it but it's more about like what tools you think are missing and you think we will appear for helping with panel UPS so what do you think about going from the title of the of the panel like what's the future in that sense of amounts um uh the future from an upset uh fluttered to be asked about that um um so for me it's always easy to to talk about something that um I don't know uh relate personally so for example if you ask me how I personally would would like to to see the future of envelopes um then I would I would say that um of course I'd love to see more more let's say common developer tools to to appear so I'll I'll explain what I mean soon right now there is there is quite a few let's say the well what which word we can use they might Ops platforms for example um so basically they offer you end-to-end um quite open-ended generally um framework how how to do things and basically you you just have to follow that that way um and if you for example change the vendor well it did get slightly different and then you have to basically compute everything or or implement or relearn um it's it's similar but it's still different so basically it's just another open-ended solution so for example I I'd love to see more tools like git for example or like terraform which which brings which kind of become the the kind of the standards for everyone to to um to follow yes you can deploy models with different vendors you can use different kind of compute and you can use different kind of storages Frameworks libraries but still um I I'd like I'd like to see more standard biased tooling um that's one thing uh secondly I'd love that tooling to be integrated into the developer environments again someone coming from this um well from jail drains from developer tooling uh backgrounds um yeah sometimes I it really hurts to see um um people being put out for a certain reason from the IDS to web interfaces uh where they lack a lot of productivity uh boosters that typically developer environment offers them so for example I mean we can just um we can at least try to agree that the coded versus this uh is kind of the defaulty of writing code so and uh there are certain let's see uh expect the level of expectations what you expect from your coded your code there should you should allow you like have this code completion Reflections code navigation at all um and then you should have some integration with Git You Know ID it should be the ID which you use normally it shouldn't be some other ideas you have to you know jump into and then your productivity just ruins You Know It uh is destroying so basically um developer environment developer activity is super important and I I'd love to see more tools uh being integrated in into that model trying to reinvent that yeah yeah so I'll be brief on this one because I want to make sure we get to the questions folks have asked in the Q a I think um maybe uh I could have already answered what I wanted the future to include by talking about uh better deployment Tools around like you know sort of realist GPU inference there's there's a lot of difficult problems to resolve with containerization for Accelerated workloads that um you know for folks like banana and uh replicate their Cog Tool uh they're working on this there's substantial improvements but um there's so there's like a pretty big Last Mile problem of resolving how to make sure that you can build on an on a Mac with an M1 uh and deploy to your uh to your you know in your gpus in the cloud um that's kind of Unsolved problem and sort of abstracting that away so that people have to think as little as possible about about the wear unless they really need to squeeze all the juice out of it uh that's something I'd really like to see um I highly encourage people to read the paper just put out from Justice foundations group at Burley for a list of not lists but two or three like big open problems about one of them um being kind of Highly precise data validation in continual ml pipelines suppose you have like a model that's retraining every five hours um and you do some sort of continual validation when you retrain but what happens when your entire snapshot is broken so your validation step passes but you have promoted a terrible bottle to production how do you catch that how do you do about validation with respect to historical partitions um how do you do this precisely make sure like you're not sending too many alerts and people fatigue I think these questions are open and right for tooling for the audience questions so the first one will be what will determine the success of ml abstracts in the next upcoming years these best practices are open source what can possibly be the competitive Advantage is our competition [Music] um like there's a lot of belief that like software is is the mode or the like tool is the mode and I think that that thing that I believed when I was still you know doing research before jumping in uh to jumping into the to work in industry but what I found is that uh you know nobody wants to actually run a database or rather people want to run a few databases as they can uh and so what it what that kind of looks like is for model Registries for experiment tracking for monitoring and like ingressing all of the things that are happening in your models all those problems are resolvable with databases and no but nobody wants to be on the hook for all three of those on top of everything else uh for much the same reasons we don't run our own git servers uh and the competitive Advantage I think the the way I would like to see things work is these are open and anybody who wishes to run their own database to solve a particular ml Ops problem is for you to do so but most organizations will choose to run uh to get a managed version or um or or a white glove version or something like that 100 or a trillion try to handle this question foreign [Music] [Music] but certainly the the the tool should clear the value in the end so basically we can measure it different ways uh but in the end it should create value for for people people should uh find this very unique um that's one way or they should be so great demand and it's hard it's so hard for people to to get this value from from others in that that quantity so in that case typically well um an option to give follows that that group need and for example we can see that right now in the envelopes um sorry not the metal Ops but machine in space for example there's a great demand for Transformers right it works so great um and and people are looking for ways to use them somehow and and also experiment uh tracking uh people it is just so basic need and people need that in so great um so high amount but even though there are different vendors right now people still did the adoption comes yeah that's my comment um and now a question about adoption so um the questions have been using old school tech for our ml projects and trying to adopt newer and more ml episode and focus ones which currently photos of using the vision about when setting up our first version basically what are the common pitfalls when adopting an upstick and practices and we can begin with Android yeah maybe I could just quickly comment on that um and probably till um be applicable not just have machine learning but in general um there are terrible things to to keep in mind like for example one is like what is the purpose right uh one purpose could be learning uh learning different Frameworks different new Concepts new uh new practices [Music] um um that's a good reason and that's where you can explore things and you could invest in exploring things um and um let's say uh yeah your dream by that objection uh sorry um the this goal so you should optimize your process for for learning basically means that you you should uh aim for trying several things not just you know invest into into one or um if for example if the purpose is different then for example your purpose is to set up the approach process up to up to speed you know and then get your project started then of course you should optimize for for that and probably you should um find some compromise between try and trying most most people and most uh um and then also time that you can invest and maybe to add on top of that sometimes people underestimate um how much time uh it might need to support certain technology um so for example certain technologies have to require you to rewrite your code for example which will Implement to implement Your solution so and uh it's the biggest Pitfall when you invest into that so and then you basically have a lock in into that technology and then you basically think twice whether you have to basically drop that technology if you don't even like that but you cannot do that easily that that would be a problem just to keep in mind yes in the I think one of the most common issues I see when people are adding more tooling to their projects is kind of like a pendulum swing hard from no tooling to Too Much tooling um they're like okay I gotta have a feature store obviously um and I gotta have my model registry and those who've got to talk to each other and like all of a sudden it's you know three months later and you've implemented a lot of tooling and no machine learning uh so I would say like it's kind of inevitable uh and I think you know I had a similar Journey with testing where once I discovered testing I was like this is the greatest thing and slight spread to Everything 100 code coverage and took me a while to realize that there's a there's a happy medium uh so there's kind of like a Natural Evolution there um you can try and get out ahead of it a little bit in your in your organization when you're setting up your first project by trying to add tooling gradually or um you know or always set you know aggressive deadlines on actually surveying on actively uh getting stuff out there um but to a certain extent you just gotta experience what it feels like to be held laid down by your belt of tools uh and slowed down by it rather than accelerated by it I don't have much more to add I think we should move the next question okay so I guess for our final question um there are so many tools coming out from many offenders for companies adopting them they end up getting too fatigue maintain different vendors and results towards using something already supported by Cloud providers uh what do you consider will happen in the ml obstacle system in the future so you think there'll be more integration sterization in the beginning she has some specific maybe it's specific envelopes right now but like throughout the history of data management we've had tool blow through multiple hype Cycles um and what happens is that the companies that find product Market that stick around in the companies that don't don't stick around so I think it will be the same thing for ML Ops why would it be otherwise um is my question I think the one thing oh I'm actually muted myself the one thing that's new here is a big shift to Cloud I think that's happened in the last few years I the implications for tooling on top of the cloud might be different um not just competing with Cloud vendors but how do you build tools that work in a cloud native environment um that's like an open question but there's no like I guess what do you consider will happen in the ml Ops ecosystem I think that those who find product Market fit will stick around yeah I I I'd agree I feel like there's sort of like periods of of explosion gold rushes um and then busts uh the uh the state of California where a lot of these vendors are located is known for a history of uh of booming and busting so um that doesn't necessarily mean it's gonna be like an AI winter or an ml Ops winter um but it just means that there will be you know consolidation acquisition Aqua hiring uh things like that as people sort of solidify around the preferred tools hopefully that means good Integrations with the major Cloud providers hopefully that means like Cloud uh like Cloud agnostic tooling for these things so you don't have to worry about whether you're doing your experiment Management on AWS or gcp or whatever um and maybe more consolidation into end-to-end tooling so like as these tools consolidate they become uh they become more like Sage maker uh so perhaps that's the end State um is for hugging face pie torch lightning um weights and biases and uh uh and banana to all resemble uh Sage maker in the end um yeah um so I agree with being said here um I would add it certainly sometimes it may seem as half there is too much you know um talks about new tools there is too too many new tools um I think that's in that's a very natural person this this is a very healthy sign um of about the ecosystem telling that something going on something um something being trying to [Music] um folks trying to solve certain problems that exist and no doubts about this these problems in in the field so this is a very healthy sign to me saying that there is we can expect a lot um of let's say a wave a new wave of of tooling so for example if we compare that with devops right um um so there's been so many jobs at this problem right uh they've been so many in-house solutions that have been puppet they've been Chef they've been ansible then they're being background and you know and then terraform and and and then still the new tools appearing uh there and then or integrating with existing tool so nothing can solve the progress that that's for sure and we shouldn't be afraid of that um I mean simply because there is no other way um sounds a little bit uh kind of great but but that's what I think about okay so thank you very much everyone um your answers have been amazing and thanks for accepting the invitation and I guess that's it for today