Building Machine Learning Models into Docker Images // Luke Marsden // MLOps Meetup #76

Key Takeaways

i'm going to sing you a little intro i'm going to sing everyone a little intro this is the back story of the emma loves community i should have wrote down lyrics but i'm just going to improvise it as we go you can hear the guitar all right yeah it sounds good last year the end of april luke said to me hey man you might wanna sit down and grab yourself a beer what i'm about to tell you is not gonna be so easy [Music] you don't have a job we're gonna have to shut down this company [Music] this time it's us but the beauty of this was that lou decided to start a community right before [Music] so i'll give credit where credit is due this community belongs just as much to me as it does to you luke was the first guest we ever had on the meet up and he said um so much the youtube comments ate them alive i've heard it's gotten better and he decided to get a speaking coach so now we're not gonna have any problems today duke i'm gonna hand it off to you this is your thank you luke for joining us today it's such a pleasure to have you here oh so many different ways and although you ended up having to fire me a year ago i'm so happy that we had a moment to work together cause you were the best boss ever [Music] [Laughter] and that my friends is the intro for luke marsden who is the founder and idea man behind this community so if you have enjoyed any of the meetups or coffee sessions at all over the past year and a half you can thank him wow that was incredible thank you dee um you thought oh let's just tell everybody just so you all know he was gonna try and put some funny words in his bio so that when i sang it it would sound all weird but i told him no no you get a special intro you don't get just the bio singing that i normally do you get something spot on that is a little closer to home anyway go ahead share i think everybody's here for now or at least most people have showed up and they're probably sick of hearing me sing but that's what you got to deal with [Laughter] uh thanks demetrius that was amazing um yeah and i mean most of the like 99 of the credit for the community goes to you and your incredible hard work both at the end of um that mmelops company you mentioned and uh and in the the time after that so so thank you um cool so i guess i'll just jump in i i want to encourage people to ask questions throughout this um because if i really want to bring everyone along with with with me on on the journey through this uh content that i've got here so if people are confused or if you've got questions please drop them in the chat um and we can we can address them as we go so yeah i'm going to talk about building machine learning models into docker images it sounds like potentially a simple topic but it actually has some complexity that's quite interesting so i'll start by talking about kind of the simple case and then talk about some of the trade-offs and some of the aspects of the kind of social aspects of getting machine learning models built into docker images in in a company where there's maybe a machine learning team and a devops team and looking at how the ml team and the devops team can can collaborate as well as possible so yeah what i'm going to talk about is to start with the simple stuff like what is a machine learning model or what is docker why would you want to build a machine learning model into a docker image and then i'm going to do i saw on the on the community slack that people really like live coding so i'm going to try and do as much live coding as i can slash demos so wish me luck i don't have backups for most of the demo videos so please pray to the demo gods for me um so we're gonna do a simple the first demo will just be a simple docker build of a model and then sending a request to it so we can do inference i'm going to talk a little bit about this question of to bake or not to bake that's the question in particular what do i mean by baking models into into container images and why why is that even a question and then we're going to look at this and then we're going to do a demo which shows how baking can go wrong uh or rather sorry that not baking can go wrong so the opposite of baking is downloading and i'll talk about i'll explain that in more detail but i'll show a demo of how that can fail spectacularly and and thereby kind of show my bias on the answer to that question um then we're going to look at some security issues around building containers inside containers and then i'm going to share a demo of some technology that i've been working on with the modzi team that's called chassis and we're going to look at how which is fully open source and devops teams are very welcome to to take it and build on top of it if it's helpful for them that we'll automate baking containers from machine learning models in kubernetes and deploy them to care serving then i'm also going to talk about the the open model concept which is this idea that why not build container images that can run in more than one place because if you do that then you can avoid you can avoid lock-in and then i'm going to give a quick demo of deploying the same model that we built in step three to a different platform in this case the modzi platform and then um there's just a piece i want to finish on around how devops and ml teams can can collaborate best and and what can we do to help devops and ml teams collaborate better because ultimately what everyone is trying to do is to ship models into production quickly and safely and and create business value from from ml and if you can't get models into production quickly and safely then it's very difficult to do that so yeah um that's what i'm gonna try and do today and yeah any questions please hit me up as we go through so let's start by just talking about the basics so what is a machine learning model and what is docker and a docker container um probably most people in the room know what a machine learning model is but i'll just give a very kind of simple description of that so a machine learning model is really a piece of data which is the set of weights that are associated with with some neural network or other parameters in general it's not just neural networks but that any machine learning model will have parameters that those parameters are kind of how the model gets trained and in the case of a neural network every neuron in the structure the structure of the of the network is kind of fixed but then it learns during the training phase what the certain weights are and so the weights are a way to take a given network architecture and and specialize it so that it's it's been exposed to some data and it's learned from that data and so the actual machine learning model kind of from a from the computer's perspective is a file or a set of files that have these weights in them that together with the network architecture um uh can be used to to actually run the model and kind of start it up again and then send more data to it to predict the the outcome so yeah the it's the these weights files plus files that describe the actual structure of the network um or the model so it's a bunch of files on disk basically um and it represents kind of the the train model and then okay so that's that's a machine learning model well what's a what's a docker container um well docker is a tool that helps people to capture the state of effectively the state of a running linux system so if you have a computer running linux then you can install a server on it like a web server or an api server that has a machine learning model behind it or something like that and you can expose that server on a port for example but what docker does is it kind of it creates it allows you to create like lots of little read-only linux systems so it allows you to kind of reproduce the [Music] it allows you to reproduce a running thing both on your laptop when you're developing it or in in development and then ship exactly the same thing to production and know that it's exactly the same thing is going to run when you get to production and so daca has taken off and become very popular because it helps to solve the problem that otherwise you might have to manually manage a bunch of servers and manually install dependencies on those servers and uh and so on and so it's become kind of almost a de facto way of building and shipping and running machine learning model machine learning models and the servers that serve them by kind of packaging them up into into docker containers and so what this thing here is the the kind of the blue layers in this diagram are um how docker images get made up so so when you're creating a docker image you start with the base layer and it's kind of everything is sitting on top of the kernel and you can think of docker containers like lightweight virtual machines where all the uh all the containers share the same kernel rather than [Music] rather than having a an individual kernel per vm um that's not necessarily a helpful way to think about it if you're not familiar with virtualization but anyway you've got this kind of shared kernel layer but then you've got this base layer which could be something like a debian or ubuntu base image and then you've got these multiple layers that you build up on top of it and so the example i'm giving here is that we add a model server so something like tensorflow serving or [Music] a flask python server that that's going to respond with with certain inferences when it's given certain uh inputs and then you can also bake the model weights so the actual files that make up the weights of these neurons or whatever the type of model is and and the model architecture itself and just add those files to the docker image and then you've got this kind of shippable unit that you can move around you can push it to a container registry and then you can run that thing in production um or in development and and kind of move it around all over the place so it's a nice way of packaging the machine learning model which makes sense from a data scientist's machine learning perspective along with all of the dependencies that it needs to run in exactly the same way wherever you run it so you won't have the problem that when you actually try and run the model in production you've got the wrong version of opencv installed on the server or something and it won't work or it'll it'll behave weirdly in a way that's difficult difficult to fix um so yeah i promised lots of live coding so let's let's try and build um a model into into a container and and send a request to it i'm going to do that here using this test drive system which gives me access to a little linux virtual machine that's got a bunch of stuff installed on it um and what we're going to do we make it bigger uh yeah sure bigger there we go is that okay yeah uh yeah i think so if i make it too big then i won't be able to see it with my eyes that's good or i won't be able to see everything i need to see okay um so cool um [Music] sorry basically what i've got here is just need to install them okay um so yeah basically what i've got here is a very simple app which and this is like the super simplest way that you could serve a um a machine learning model just by having a flask app written in python that is a an http server like a rest server that opens a model file when it starts up and then it calls model.predict with some post data whenever uh whenever it receives a post on slash and so so that's fine um the other thing though is that we need the actual model file itself so that model.pickle file and that one is is here so i pre-trained this model i won't go into all the details about exactly how it was trained it's just a psychic learn model um that was trained quite simply i think it's trained on the diabetes data sets i think it tries to predict whether you have diabetes or not um and then i also need and i promised you live coding because this doctor file that we have here isn't actually quite right um but let me see resizing the window kind of screwed things up so i'm gonna try a different way of zooming in how about that um so what i'm gonna what we what we can see here is this is a version of of a container so let me just describe um kind of what what we're looking at um we've actually got yeah so so this is a docker file and so if you're not familiar with docker files docker files are basically a list of instructions that tell docker when it's building one of these images what to put in those base layers so if we go and look at you know this picture here each line in this file basically puts a new layer on top of this container as you're building it up um and so if we go back over here um then we can see from python 3.9 is the base image that this thing is being built from um then it sets it worked says run everything inside slash app inside this container so kind of make that the working directory and then these copy commands are pretty self-explanatory it's like let's copy the requirements.txt file that we have here into into the container image then we run pip install on the requirements.txt and then we copy the code that we want to run in which is the serving code that i showed you but what we haven't done yet in this in this image is to actually copy the model in so let's do that the special lines that we need that are here so that's basically two more layers that we're adding so we're adding this histogram file don't worry about what that is uh the the main event here is that we're adding the model.pickle file and we're putting that in slash app slash model.pickle and that's where the um that's where it's going to get looked for um and then we can just build the container image um so docker build dash t takes the name of a tag which is um kind of the name that you give to the image and then dot just means use the the current context so the current directory that you're in uh to build the container image um or for things like when you have this copy directive um the dot just means copy things basically from the current working directory so we see we've got this model dot pixel file here um so we built a container image um and now we can run that container image and that's now running in the background and i can now send a request to that container um and so i've sent in some features um to to predict whether or not uh i'm likely to have diabetes i guess and the model server responded with a result which is 106. i don't actually know if 106 is a good score or a bad score for this model um but that shows that it's working so that's kind of like the basic the easiest way that you could build a container image or build a machine learning model into a container image and that basically that would be to to learn how to use docker to figure out how to run your model in production so to provide some code like this serve.pi file that that responds to http requests and then to create that container image yourself and then i guess run it in a kubernetes cluster or ask a devops person to run it for you or something like that um so yeah any questions about this kind of base no pun intended base layer of understanding around around what we're doing with machine learning models and how we're baking them into or how we can put them in container images the chat is pretty quiet at the moment cool feel free if anyone has a question also along the way i can stop luke if you just post it in and then we'll have him answer it oh here we go we got one right on do you containerize for batch predict 2. yeah i uh you absolutely can yes um so this is an example of a of a rest uh interface the um actually the so the the platform that uh that i'm going to show this integrating with later this platform called modzi is kind of a batch platform so you can upload a large chunk of data to it and it will chunk through it and and give you the answers and um the way that that's implemented is that each um kind of component of a batch gets sent to the containerized model that's containerized in much the same way so yeah you can take the same approach and run your batch operation through just effectively sending multiple api requests um so so yeah the mods is a good example of how you can do batch prediction uh with containerized models um and i've seen that be used elsewhere as well so i hope that answers the question sweet cool i'll let you know if anybody else has one thank you so uh to bake or not to bake i want to talk a little bit about about the pros and cons for baking but first i'll describe what the heck do i mean by that so what we just saw was taking these model weights and putting actually baking them into the container so there's a layer here that has this model weights files inside the container itself now there's an alternative to this which is somewhat popular i think it's kind of about 50 50 in the community as to whether people are baking versus downloading the alternative is to have the model server when it starts up download the model weights from somewhere by an environment variable uh when the model server starts up now for example you could write a piece of code in your model server that says go and download my model at startup from an mlflow server based on an environment variable that i specify now that's actually has some complexity because what you're doing there is you're you're adding your ml flow server to kind of the surface area of the infrastructure that has to be running in order for your model to be running so basically there's like an operational challenge there is that if you go for this downloading approach then wherever you download your model from um the it has to be online at the point at which your model starts up in order for uh for everything not to break so so that's kind of one of the challenges with that and um and i'll demonstrate that in a moment i guess another challenge is uh or but on the other hand the benefit of of not baking your models into your container images is that you'll just end up with far fewer container images because you can have like a generic container image that knows how to run a pie torch model or something like that as long as it has the right dependencies in it and then you can reuse that with multiple versions of the same model for example just by changing that environment variable which is you know where it where it downloads the model from now um there's another worry around not baking which i have which is that if you don't make your i've seen a lot of people basically put their models just throw their models on s3 that seems to be quite a common pattern which is you put the model weight files on s3 and then you you deploy your model servers that download the models when they start up everyone's happy then your deploy to production step can be that some person on their laptop uploads a new version of the model file to s3 and then you restart all of your containers now that has alarm bells going off all over the place for me because you are basically changing which version of the software is running in production but not recording which version was running at any one time and so there's a like when people take that approach i feel like then you're really asking for trouble um so i'll talk about that a little bit more um later on but um yeah i think like i mean you can kind of tell like i'm biased towards not towards making being best well i love that how you're framing it too and saying like you're setting yourself up for disaster there when you do it like that and i mean there is a reason i mean what i really love about your background and your knowledge is you have all this devops best practices just like in your blood right so when there are things that come up in mlabs you instinctively know like that could be a way to that's a recipe for disaster right there frankly yeah yeah we've got a few questions coming through that i want to ask you before we jump into the demo uh durga is asking what's the best practice or design to follow when api serving should have fewer latencies for example if prediction response should be less than 10 milliseconds yeah so the first thing that comes to mind is just making sure that you're not doing anything like loading the model every time you process a request so the model server should load the model and initialize it if it needs to be initialized when the server starts up and then make sure that it doesn't try and initialize or load the model every time it it responds to a request but just reuse that cached model object to to send requests to another kind of another aspect to this is well you can only go as fast as the model itself so there's kind of the stuff around the model um but then like the model server but there's the model itself and if the model itself is is just slower than 10 milliseconds at processing a certain type of data then there's nothing you can do about that um apart from well so firstly you need to make sure so you should benchmark your models when you're developing them to check whether they can actually meet your kind of production sla and then the other thing would be that you probably can speed your models up in production if you need to but that might be a matter of running them on gpus in production and stuff like that um so then um yeah making sure that you've got access to the right compute and configuring your cuda drivers and stuff which is a whole another talk um then the the other piece that comes to mind is making sure that you can deal with parallelism so oftentimes a python server might be kind of a naive implementation of a python server we'd only be able to process one request at a time and so you might find that your response response latency is fine when your service is quiet but it goes through the roof and you get horrendous tail latencies when when you have concurrency you have lots of requests at the same time and so there's a couple of ways of dealing with that um in python you can do multi-processing i mean that's basically um what fast cgi was designed to solve back in the day and there's a bunch of different approaches around that so yeah just making sure that you can deal with with parallelism or multi-processing um which can be harder when you've got gpus because you might not want to uh might not be able to use them in parallel if you run out of memory and stuff so yeah there's kind of those are the topics that come to mind around that i hope that helps sweet so another question coming through i noticed that you have several copy directives in the docker file is it possible to combine them to reduce the number of layers in the image and thus the image size yes so i think there's like a secret feature in the docker file format which is that if you i forget if it's copy or add i think it's add and it's not the other one which is confusing but if you add a table then it adds the contents of the table to the image rather than the tarball itself which is like a bit of a weird feature because it's really implicit and a little bit surprising and i think that's why they implemented copy because copy doesn't have that weird behavior but you can use that behavior to add multiple files in one go if you want to and similarly if you're using the run command then you can chain your run commands with lots of ampersand ampersand backslashes which you'll see all over the place in docker images in docker files to to reduce the number of layers nice so last one before we move on eddie's asking can you discuss more on your views on the trade-offs of to bake or not to bake question in the context of model updating rollbacks for continuous learning on a related note are there tools you use to manage the timeline of swapping production models yeah definitely so i think the in terms of model updating it's a really it's a really good point i think that when you're doing updates and rollbacks that's when you really really want to bake because if if you've got like two different sources of truth for for what version of the software was running in production i.e you've got the tuple that is both the docker image that you used and the uh and the the model files that you had on your s3 server or whatever then actually knowing that you've updated the right thing or that if something goes wrong that you've rolled back the right thing and that you'll make sure that you're keeping a record of both of those things rather than just the one of them um it's just going to make it harder to keep track and so there's fewer moving parts when you're baking something and you don't have this problem of like oh am i running the right version if i need to update the model server am i running the right version of the model server with the right version of the model and so on um so yeah definitely in terms of rollbacks it's uh rollbacks and updates that's where baking helps you be more confident that for example if something goes wrong then you've rolled back both the version of the libraries and the version of the model to the previous known good version rather than like maybe in a in a panic like rolling back the model file but not the version of the server and then everything might be down so like um that's uh i think that definitely skews towards baking making it easier from an operational perspective um in terms of yeah managing the timeline of swapping out production models there's a whole space of tools around continuous delivery and so if you google continuous delivery tools then then you'll get a bunch of them my favorite approach is called git ops and those tools like flux and from wave works and argo cd which allow you to use a git repository as a source of truth for what should be running in production and then there's a it's like fluxes or argo cds responsibility to reconcile your cluster to the the state of truth that you're asking for in the um in your version controlled yaml or whatever so um yeah definitely look into continuous delivery tools and there's a whole space of like a b testing and rollouts and stuff around that um which i probably don't have time to go into for another day yeah so i guess yeah let's keep moving i posted the demo link in the chat if anybody wants to follow along cool yeah and to be clear there's a couple of different demos that uh test drives that i'm using there's the box kite one which is a a different um system that kind of shows the the non uh sorry the the non-baked approach which uh we'll talk about in a minute and then there's also uh this chassis.ml link um so if you want to try out the test drive for uh for the piece that i'll show you later then then it's on chassis.ml so yeah please uh post that link as well um so yeah i i'm actually conscious that i'm probably gonna run out of time if i don't speed up a bit so let me show you two more uh quick demos the so this one i i want to show you the weakness of of not baking your models and um i was going to try not to be too biased about this baking versus non-baking thing but i think it's pretty clear that i am very biased [Laughter] like don't don't download your models at startup it's a bad idea um so yeah let me show you let me show you this piece so i'm gonna let me just find the instructions here because it'll help me stay on the right track um so yeah this is a kubeflow ml flow setup um that i'm gonna show you quickly and i have to use the right thing okay so here [Music] so i'm just going to spin up a so lots of people these days do data science work in an environment like this which is inside a kubernetes cluster with something like kubeflow and in this case we're using kubeflow notebooks so let's see um i'm going to just spin up a quick kubeflow notebook server which is a jupyter server i'm going to use this custom image i'm going to turn off persistent storage for reasons i won't go into and then configure it to talk to mlflow then i can hit launch and then we can run through uh run through this notebook so this is yeah this is an example of a system which downloads models from ml flow when it starts up and i'll actually show you the code quickly for that the box type project by the way is is actually about um model monitoring but i'm just using it it's example because it happens to download models from ml flow um at startup so [Music] if we look in this app here oops i need to go into the kubeflow mlfl one [Music] yeah so this is going to load the model from ml flow so actually when you can see this is a model import time so model import time in the serve completed file which gets uh run when the container starts up um it's actually importing the ml for a client and then based on this run id which is specified as an environment variable it's going to download from ml flow into this like temporary directory when it starts up and then start serving the model so um this is like doing the bad thing that i said you shouldn't do basically so um we're going to train the model um we're going to [Music] log it into mlflow we're going to check that we can send stuff to it um and then we're going to deploy that model into the kubernetes cluster using some yaml that um the key point here is that we're passing the ml flow run id from the model that we just trained in the notebook into um into the production instance of the model that's running and then we're gonna see if it's running and then we can see yeah okay we can make predictions so it looks fine right it looks like it works um except that if the ml flow server were to were to go away like nice ml flow server shame if something happens to it um if the ml flow server were to disappear then everything would look like it was still working until like kubernetes automatically scaled up your deployment because you got a spike in load or one of your servers failed and kubernetes had to redeploy the containers onto a different machine and then they would just be silently failing they would be failing to download things from the ml flow server at startup so let me try and convince you that this is a bad idea by i'm going to go in and actually break the mlflow server and then we can see what happens um i like breaking things so let's see oh i love this um so here's our ml deployment happily running la la la this is all fine um and then ml flow oh looks like you've got an my sql server ml flow um nice database you've got there it would be a shame if something happened to it that's some serious foreshadowing what that's some amazing foreshadowing so maybe you maybe assist admin accidentally deleted your mysql database um or something else bad happened like my sql databases are kind of fragile right you don't want to have a mysql database at the heart of your uh of your of your production system unless you're doing unless you're taking a lot of care to look after it um and the point there is that um yeah like as soon as you use this approach to to baking um sorry to downloading your model when you start up your container image then you're putting all of the ml flow or whatever you're using for model for model management into your uh into the production into the the scope of the production system that from a devops operational perspective that you have to manage um so let's log out of this my sql server now um and have a quick look at um these container images so as i said they're still running because they only download from ml flow when they start up but um if we delete the pods then um kubernetes is going to restart them because kubernetes is good like that it's going to try and keep the system up and running but i delete all three of them it can you make it a little bit bigger yeah sorry oh i think so no i can't because when i try and oh you're already oh okay no worries yeah oh no there we go that's okay so yeah basically we're in a world of hurt now um because uh these containers when they start back up they go into error states um and let's see did i break it enough that it definitely stopped working yep okay so now your ml model in production is down and the the scary thing about this is that it happened sometime after we had the problem with the database so it's not even going to be easy to figure out like what happened um because it was only when kubernetes started restarting these services so basically this is just a warning like if you're going to not bake your models in and download them when you start them up then a you have to be really careful to um to not mess up immutability and reproducibility in terms of just like uploading a new version of the model on top of the old one and restarting things as a way of upgrading them which is bad but you also need to start to operate your ml flow or whatever you're using for model management like it's part of your production system and it kind of brings that into uh in into the production so yeah wouldn't because i mean you have prometheus and grafana as part of this right like wouldn't that have caught it oh yeah i mean you can set up monitoring but for your production system and you should um but you're still gonna have to like go in and do something about it um it's it's still like it's it's even more operational overhead to say okay well i have to set up monitoring for my ml flow instances one of the problems is that ml flow instances are often just set up by the ml team and they run kind of for the internal purposes of the ml team um and so if they're not centrally managed by a devops team then devops might not want to introduce a dependency on them for production yeah so like benefits of baking model management doesn't become part of your production system you can still serve inference even if someone's messing with your ml flow instance there's fewer moving parts and higher reliability basically um so um that should have being like that sorry um when you're doing the downloading approach there's no guaranteed reproducibility of inference results you do however get fewer overall container images like if you have like n different models and m different um [Music] sorry if you have m different models and n different kind of model servers or model server versions then you end up with just n container images rather than n times m so it's like takes a quadratic thing down to a linear thing which is good but um it's actually not really a problem to have lots of container images because container registries are built to be able to deal with them and they're just backed on object storage um the other another benefit of downloading at startup is that there's less data if you've got very large models like models that are gigabytes big then you don't have to put those in your container registry but then you do you have to put them somewhere else like you have to put them in ml flow and ml flow is back to an object storage as well so it's all just object storage at the end of the day and container registries can cope with large uh container images so yeah i think baking models is better hopefully i've convinced you um so i then want to talk and i know that we're we're actually getting close to the end of the hour so we've got kind of 10 minutes left i i'll go quickly through the rest and then maybe we can go five minutes over if that's okay um yeah that's fine with me so i wanted to talk about the security challenges of building a container inside a container so this talk wouldn't be complete without a meme so here you go here they have a meme one does not simply build a container inside a container what i mean by that is that so for example what we saw just there was my kind of the working environment for a data scientist was this jupiter notebook that's running inside a kubernetes cluster because this jupyter notebook is running inside kubeflow notebooks and so i mean i can try like how's that i don't have docker installed um docker doesn't run nicely inside a container and because you're using kubeflow notebooks here inside um kubernetes it doesn't uh it doesn't work very well if you try and layer things like that so um basically yeah like data scientists are often work doing work inside a kubernetes pod these days even if they don't know it um or you might be part of a team that's working on building an internal ml ops platform and you need to to have a reliable way to to solve model builds at scale um it's generally a bad idea for data scientists to do anything on their local machines if you have workflows that involve like alice or bob the data scientist actually building the container images like i showed you in the first demo just on their own machine and then pushing it somewhere then what happens if their laptop gets stolen what if happens if they leave the company and they had like some special stuff on their machine that was how they built that model so you kind of left you kind of running out of options for like where can we run this this image build i can't i don't want to run it on my developer's local machines i can't easily run it inside the development environment that my data scientists are using um and even if i did want to run kind of docker from inside kubernetes then you have to do this thing that's called bind mounting the docker socket and this is something this is a technique that gives security and compliance people um like a big headache and because it effectively gives everyday scientists root on the machine that's running the cluster and that's a very bad idea so google solved this with an open source project called canaco and it happens to be the same same open source software that powers their cloud build service so they have a cloud builder as part of google cloud um and i won't go into all the details but basically cadaco just doesn't try and do the clever like overlay fs stuff that is kind of a performance optimization to make docker work instead it just when you when it passes a run command in a in a docker file it just runs that command inside the container that it's already running in and then at the end it ties up the entire file system inside that container and sends that off to the container registry so this can eco thing is like part of the solution here it feels like but canaco itself is a building block um and what you can do with canaco and what we did with kaneko in this project i'm going to share in a minute is that we built a centralized service for image builds that run in a secure way on these multi-tenant kubernetes clusters um so yeah that's basically why you shouldn't try and build a container inside a container um so that's what this chassis project is that that we worked on it builds existing ml flow models into these container images um and it also exposes this python sdk which means that data scientists don't actually have to learn docker they can just use a python api basically and the way that that works is that the data scientist will publish an mml flow model into the chassis using chassisml.publish so it works nicely with with ml flow and then that will make an api call to the chassis api server which is this open source python api server we developed that starts the canako job as a kubernetes job which builds the container with the model baked into it which um is this the model itself is this diamond and then the is this a trapezoid anyway the trapezoid is is the container itself and that that container gets sent off to the container registry or can be pushed into the mozzie platform and it will work in in both of those in both of those contexts so yeah i would take you through a quick demo of that um and and then i think we'll wrap up because we're gonna run out of time so i've got another one of these test drive setups and in this case we're not running uh kubeflow um and everything on here it's just a kubernetes cluster that has um [Music] i'll show you what's running inside this one so it has the combination it has the the chassis service sorry um which is that api server i just pointed out um it also has an object store um implementation running on it and it's also got kf serving which is a system for deploying models into and actually running them in production so i'm going to quickly show you that then um [Music] so basically and you can try this demo yourself by running through the uh the service demo notebook first and then the kf serving dot ipanb notebook second and like i said this is all available on chassis.ml so um yeah in this service demo i also need a docker hub account so i'm just going to sign into my docker hub account quickly i'm going to copy and paste this because i need it later so [Music] we're going to train a model um this is an mnist model uh and we're going to also wrap it i can make this a bit bigger um we're going to wrap that model with some custom pre and post processing which is a nice feature of ml flow um that you can you can basically serialize the serialized and custom logic along with your actual model you can then define a condor environment so just this is a nice way of baking specific dependencies that you need like pie torch um or whatever uh opencv into your container image and they will those dependencies will get installed when the model is built rather than kind of when the model starts up so it gives you better reliability about making sure that those are working reliably then we can save the model into this ml flow pi func format load in that model and test it and then we can see we've got a bunch of files we've got the pickle file a conda yaml and an ml model file that describes it and then we are going to log into docker hub so let me just fight with my password manager for a second so i'm just going to put my password in here because then the chassis service is going to build and push the model with that ml flow model that i just built and pushed into the ml flow registry that exact ml flow model is going to get baked into a container image and automatically pushed to docker hub for me so kind of from the data scientist perspective they didn't have to think about docker they didn't have to think about um the the server the model server that they put in here they can just they can just go ahead and publish their email flow models directly into docker hub basically so then i can show you what's happening um the chassis service that i described here um is running this kubernetes job and the kubernetes job is here and so we can see what it's doing is it's installing all of those dependencies that i mentioned um that are specified in the condor environment it's baking those into the container image it's also baking in the model files themselves and then it's going to go ahead and push that model into into docker hub and then the second part of the demo that i can do quite quickly hopefully um is deploying that into kf serving now um i'll just find the tag so that's the image that just got pushed by chassis into um in there and noticed by the way that yeah chassis built that image in a multi-tenant kubernetes cluster without causing a security headache so you didn't have to think about about that um and now we're just passing the image tag into into this kf serving setup um we're not passing in a reference to downloading that image out of ml flow um and now i'm just quickly going to [Music] run some commands in here that's going to keep going in the background so we're gonna deploy that model you have to wait for the pod to get created uh so that's creating the container so what's happening there is that that's downloading um the container from docker hub um that normally takes about 60 seconds so yeah any questions um while we wait for this to ping over to to ready there is one from a little bit further back that was a from jess general question assuming our container is part of a larger system and its output is effectively our contract between other systems how do you suggest testing and managing that contract so that other teams are comfortable integrating your model into a production system yeah i think that's a really good question i think the answer is in the same way that you that you test any um any software so the there should probably be acceptance tests that deploy whatever the kind of candidate version of your software is that tests them against other other components in your system and you can take kind of a unit test approach so yeah basically acceptance testing is and and you should you should try and automate that as as much as possible um so yeah interesting okay i think i lost the um the connection here so i'm gonna have to spin up a new one quickly i'll just do the kf serving piece here so i've got too many jupiter notebooks up now oh no classic so we're going to deploy that thing and then we're going to go into the ssh for it oops around that didn't i so now we're going to wait for that thing to start up um okay we're back where we were sorry we're going to go a few minutes over today i think um so yeah it should take about 30 seconds to to start up that container so so yeah that's the container that automatically got built by the chassis service got pushed to docker hub um and then we're deploying it into kf serving okay there it is it's running and then we can get the hostname out of it then we're going to download some some data that we use to test the thing and yay we got predictions out of our production model um and so that was all done without the data scientists having to build a container image themselves they just use that python sdk and so basically the what i'm proposing here is that if you're an mlt if you're a devops team that has an ml team in your company then consider building a centralized service for model builds that you can provide as a service to your data scientists so that they can do the model builds themselves this will make you very popular in your company basically um because it means that um basically it solves this problem of like if the ml teams are the teams that are that have to build the container images then it puts this onus on the ml teams to ship production-ready software um and to have all of this devops knowledge that you might not necessarily want your ml teams to need to have whereas if you make the devops teams build the container images and the you know like the ml teams email them the model files or something terrible like that or slack them to them then this is going to slow down shipping models into production because you're going to have to have multiple teams involved in in shipping something and that just means that um things that latency increases as soon as you add multiple more more people to to a process so consider building these consider creating this internal model building service that your data science teams can access via a python sdk feel free to build on the work that we've done at chassis.ml and that way the ml teams can build their own containers but do so in a way that as a devops team you can customize and control um and which are production ready so um yeah i'll see you in my talk six minutes late i wouldn't expect any less from you actually [Laughter] i just want to mention that people want to get involved in open source projects and you do a lot of open source projects you have open source basically in your blood and so we have a few of those in the community as like basically community projects almost we could call them that i haven't cleared that with you but i would call them a community project like the ml ops stacks the chassis is a great one the other one uh box kite you've got all these different projects and so if anybody wants to join you in creating cool stuff we can make that happen i would say reach out to you on slack right and then they can figure out how to best be involved and from that i guess that's all yeah if anyone else has any other questions again reach out to you on slack and we're going to finish it there luke as always man it's a pleasure i'm sad that we can't do this more i used to chat with you once a week now it's more like once a quarter but every time that we do talk it is awesome really cool to hear your wisdom what you got to say i really thank you for this and the voice coach paid off you didn't say um as much so hopefully the youtube comments do not eat you alive well kudos to you because the truth is i didn't get a voice coach but you were my voice coach just by saying that at the beginning you made me think about it [Laughter] i guess we wouldn't even call it a voice coach a voice coach is what i need for my singing in the beginning and you speech coach is what they would call it so anyway that's it i'm sure everybody is late to their next appointment thank you all for staying with us join some open source projects with luke and the team and we will see you later have a great week everybody bye thanks luke you