DevOps for ML with Dotscience - Democast #1

Key Takeaways

all right everyone welcome to one of the very first swim old demo casts this is a little experiment we're running we're in addition to me and a wonderful guest in this cake in this case Luke Marsden CEO and founder of dot science we've got a demo that you can catch if you're listening to this via the podcast I encourage you to jump over to YouTube you can find this at some old optimally i.com slash demo cast slash science feel free to continue on audio only but this conversation will be enriched by a demo that we'll be jumping into before we get too much further and Luke welcome to the demo demo cast a sound thanks I gotta get used to it yeah thanks so much for having me on awesome so you know let's get started having you introduce yourself to the audience tell us a little bit about your background how you started working at the the confluence of ml of step ups and machine learning absolutely so hi everyone my name is Luke Marsden and the founder and CEO of this little company called science we are working on ml ops and making it easy for data scientists to deploy and monitored machine learning models on kubernetes I guess my background is very much I've come from the DevOps space as have many of the people on the team and I was deeply involved in the early days of docker and kubernetes where my former starter cluster HQ built the first data layer for containers so connecting docker containers to persistent storage for example and so I'm coming at the world of AI and machine learning and we all are we're coming at the world of AI and machine learning very much from a DevOps background and what we found is that a lot of the things that software developers take for granted in terms of the tooling and the workflows that they have which enable controlled collaboration and continuous integration and continuous deployment the those tools and processes aren't as mature in the world of machine learning and so we see a lot of chaotic practices around like emailing and slacking Jupiter notebooks to each other manually keeping track of metrics and hyper parameters and basically failing to have reproducibility and provenance in the same way that people take for granted in software engineering and the reason for that is that machine learning is fundamentally harder than Software Engineering so that's where we're coming from and that's what we built the dot science platform at all nice you know I'm curious when you say machine learning is harder than software engineering I'm curious what that means you I imagine that not many people listening to this show will disagree with that but I'm curious why you say that yeah sure and actually if you don't mind I'll collapse and a slide that I you used to describe that problem in particular and and what I mean by that is that there in the software development lifecycle we have this code test deploy and monitor process and and that of course has its complexities and its subtleties but fundamentally that is what software development and DevOps with software is about but when you start thinking about models as software and when you think about machine learning models as deployable artifacts then you realize that there's more complexity in that space because there's data and there's models as well as the software which trains the models and when you've trained a model then you get metrics coming out of the model as well and so you've got lots more moving parts basically you need to be able to keep track of the data which version of the dataset you train the model on you need to keep track of which parameters you used to train the model and all of these go into what called data runs and model runs so data run is where you're tracking the relationship between some input data and an intermediate data set so for example you're taking raw data and you're turning it into a training test and validation set the training a model and then what we call model runs are where you are training a machine learning model on a certain data set maybe that intermediate dataset that was the output of the last step and and then you're creating a model and it's that artifact that's deployed into production and then monitored and it's just it's more complicated there's more moving parts and if you're not careful it's quite easy to get into a mess nice nice Before we jump to deep into this and transition into the demo that science was a founding sponsor for our recent conference from akane i platforms uu team Nick in particular presented on ml ops generally but also a manifesto fit you are evangelizing tell us a little bit about this manifesto absolutely and it was a real privilege to be a founding member at the the Trimble con it was a fabulous conference for us really kind of AI platforms is obviously what we do so it was it was it was a really relevant conference for us and and thank you for hosting it and um the the manifesto that I'm talking about is is on the website here and it basically says that in order for machine learning as a discipline to to be production ready to be mature and to to get to the same level of sophistication that software engineering and DevOps does then you need four characteristics it needs to be reproducible accountable collaborative and continuous by reproducible I mean that someone else has to be able to come along and reproduce the model that I might have put into production six months ago even if like I've left the company for example accountable means that you need to be able to hold the model to account in the same way that you hold humans accountable for their decision-making which at least means knowing who they are as in what which model it is which version of the model and on what basis it's making its decisions and by that I mean which data it was trained on at least collaborative is that is about being able to collaborate between multiple people without treading on each other's toes in the same way that github has enabled asynchronous distributed collaboration for software engineering and continuous is really about being able to deploy a model automatically and then statistically monitor it for drift and then take that background into into the model development lifecycle again so so that's why we're coming from with the manifesto and yeah it's it's proving to be something that's resonating with with a lot of people um I was I was talking to a guy in Japan just this morning who saw the manifesto on Twitter and reached out and wanted to connect so it's definitely working it is there a particular challenge among these four that resonate most strongly with people I think it depends where people are at in their lifecycle I think that for a lot of people a lot of especially enterprises they're really just struggling to get models into production at all because the whole idea of a model is software can be alien to some organizations and and even then the the barriers to getting those models deployed into production can be significant just because they're new and different organizations are familiar with with shipping software but this is a new world and so I I guess that that sort of highlights the continuous aspect of the manifesto continuous delivery or at least let's to ship models at all there's kind of a blockage there but then more sophisticated users who have got models into production then find that they're coming across the reproducibility and accountable see problems and and as there's teams scale up then collaboration becomes a bigger problem as you have more people involved okay you know one of the things that that I've written about in kind of talking about this this base in particular in the machine learning platforms ebook that we published earlier in the year is there's this at least I observe a tension between kind of products and companies that are going after this broad end-to-end machine learning workflow versus offerings that are kind of specializing in a particular niche or step in the workflow that have the luxury of going a lot deeper I'm curious do you see that tension as well and and how it manifests itself for you sure I mean there's certainly a lot of product to build when you're doing the end-to-end lifecycle like we are a lot of expertise the gain and yeah a lot of use cases to see and and I think I mean that's where dot science being open and interoperable comes into play we have a little spiel about that on our on our on our product page here but I think the the key point there is that we're doing everything that you need in the lifecycle well enough that you can just pick up those science and use it out of the box for probably the 80% use case and because we come from a background of data versioning and and provenance tracking we're really strong on the on the development side and the reproducibility data versioning and provenance the pieces around deploying to production and monitoring and production are newer but we also have a lot of expertise in the team around Nettie's and and CI and continuous delivery and such but I guess what I'd say there is that if if you need to do something more sophisticated than what we currently support on the deployment or the monitoring side for example it's super easy to plug dot science into your existing CI system for example and so we're announcing a partnership with git lab in the coming weeks where we are also going to support building your docker images in get lab rather than inside dot science and yeah that philosophy of not being a walled garden means that yes we provide the end-to-end but also if you want to swap out what I don't have a sort of lightweight built-ins with something more sophisticated then we're also providing the hooks to do that so I guess that's how a tackling the the end-to-end depth versus breadth challenge mm-hmm sounds a lot like you're taking a cue from the original docker matter of batteries included a bit replaceable that's true and actually it's funny you mention that because I was very involved in cluster HQ in trying to make sure that that was actually true for storage plugins with docker and adding the right hooks in at the right time is essential and I could talk about that for a lot longer and then we have time on this podcast you mentioned that the the core of the company's DNA if leased from a product perspective is the the storage aspect and what that enables from a provenance perspective can you drill a little bit deeper into that effort yeah sure I mean it might actually be a good moment to show you rather than telling you I mean how about I kick up with the demo is that all right yeah yeah okay cool so so I'm gonna jump straight in here and I've got I've got two accounts and I uh reiterate for anyone that wants to follow along this will be up on YouTube the quickest way to find it will need to go to tremolo I calm slashed demo cast slash dot science awesome thanks Sam um yes I'll dive straight into the demo what we've got here is we've got two user accounts we've got the Luke demo 102 user account here and then in this other screen here I've got an account for to my imaginary collaborator Fred and so bear that in mind when we're looking at these screens is that Luke's got this this white Chrome browser and fred has got this bright yellow one so that's kind of easy to follow it makes it easier to follow along to to who's doing what as we go through so the first thing I'll show you is that we've got some public projects in slide dot science and they are kind of like sample projects so as soon as you go into dot science this is a brand new account I can go into the road signs end-to-end example and I can fork this project and so just by clicking the fork button it creates that fork and so what you'll see is this notion of forking is very similar to get hubs notion of forking and it does enable this sort of style of collaborative or constructive collaboration we have I formed well so you folks a project with a complete history of runs with all of the data sets attached to it at the in the same way that the data sets were attached to the original one and and any code data and metrics that come along with it now the only run in this project actually is the Run which uploaded the files originally so at the moment this this project isn't isn't that interesting but we'll change that so the next thing I'm going to do is I'm going to hit this button to start Jupiter now Jupiter of course is a very popular tool for doing data science a lot of data scientists work in Jupiter which is a an IDE basically for for doing exploratory day science work building models and writing Python primarily and we also support running dot science just via the joslyn's Python library so you can drop it into whatever other development environment you already use whether it's vs code or or they're more Emacs or whatever your preference is but mmm but yeah Jupiter is is baked in and what you can see here is that as soon as I spun up this project the Jupiter environment was in exactly the same state it was when when we published this sample project so I'm I'm just gonna jump straight in here and what we can see is we've got the dot science tab that says that science is ready and waiting and if we if we get started the first thing we're going to do is do a dot science run and what I mean by a run is that we've imported the dot science Python library and then we tell dot science you're running in a Jupiter notebook with the interactive command and then we start a run and then we say well this in this run we are going to output a file and this actually is just a file that we happen to upload into Jupiter it could be a file that we've downloaded from the internet it could be something that we've slept in by a PI spark or integrated with a version s3 bucket by adding an s3 dataset but in any case and we then do a desktop published command and so this is kind of important because d/s not publish corresponds to exactly the the point at which dot science will version things it will gues not publish will tell dot science i've done a day to run or a model run please make a record of it and so the very first thing you can see is that that we're recording the existence of this CSV file which is the labels file and then if we go to the runs here we can see that that run has been recorded and the interesting thing about this run is that we've got the provenance information for for what that run did it said this version of this code running created this version of this of this file um and we also record who did it when they did exactly what environment they did it in so the version of the libraries and Python and everything and we're also recording the exact version of the of the notebook so you can see the diff between the notebook before and the notebook now is simply the addition of this metal science run metadata which is how dot science picked up that that run happened um so let's jump back in I know we don't have very much time to jump to go through all of this so I'll go fairly quickly um the next run that we're going to do is a conversion we're going to convert that CSV file into a JSON file and that's because that's needed for one of the latest steps and so let's look at the provenance for this well the provenance for this is slightly more interesting now because we can see that it's kind of joined up this version of this JSON file came from running this run in this notebook which read in as input this CSV file and that CSV file came from this run and so we're kind of building up this map of what's happened to get us to the current state and it turns out that this provenance graph or this this sort of map of the history the lineage of all of the assets in the project is super useful when you come back to try and figure out what happened later like this machine learning model that's making decisions in production exactly which data was it trained on you'll always be able to tell exactly the answer to that question and do you only get that if you're running your pipeline in a single notebook or if you've got multiple users in a collaborative environment working on separate steps in the pipeline are you still able to get that comprehensive view of the of Providence yeah you're still able to get the the comprehensive your provenance even if it's multiple people doing different steps and you can also get that view of provenance when some of the steps happen in Jupiter and some of the steps happen via excuse me by executing a Python script through one of our pipelines so I haven't included the the pipelining feature in this demo but we can we can maybe do that in a follow-up or something so so this final run is is downloading some some data from the internet and so we can see the the provenance graph of this one and so this is reading in this well it's downloading the data from the internet and then it's spitting out a test or pickle-train dot pickle and a validation set pickle file and those data sets are going to be used later in in the example um so I'm now going to go ahead and train a roadsigns model so the data that I that I downloaded is a bunch of German road signs and what we're going to do today is we're going to build a Karass model intensive flow that is able to classify pictures of road signs and you could imagine that that would be useful in on in an autonomous vehicle for example I'm gonna start out the tool itself is agnostic to the framework yes exactly so dot science will work with with any data engineering libraries it will work with with any machine learning libraries the only limitation at the moment is that the deployment and monitoring piece which is turning a model into a docker image and deploying it to production that currently only works with tensorflow but we're working on psyche at land support right now and then we're going to add support for other ones as we get demand from from customers to do that so I've got this is there but I'll let you finish this first no it's fine this is a good time to ask questions actually because doing this image pre-processing here takes a takes a minute or two well I'm curious about you know the size from the you know your history with with docker and the team's familiarity with that that approach there are lots of different ways that folks go about kind of packaging up models to for deployment why the choice of doing that inside of a container well I think that docker containers have pretty much become the de-facto way to package up machine learning models at least that's what I've seen um the benefit of of containerization is that you can create a machine learning model and build it into a docker image and then you're absolutely guaranteed to know that that exact image with all of its exact dependencies and all of and and the model data is it's immutable at that point it's frozen and that means that if that model gets deployed into production today it will behave in exactly the same way that if it's deployed into production in six months and it doesn't depend on the version of like the Python libraries on your server at any one point in time and also that immutability is important for reproducibility because it means that if you've got a model running in production you'll be able to trace back from the tag on the docker image to the exact provenance in dot science and so yeah using using docker in that way is is a good fit for what we're trying to do in terms of reproducibility and bringing these DevOps principles to to bear in machine learning and so does that container that you create include some type of you know runtime scaffolding that exposes your model as a service or is that ourselves ya know that's exactly right so I'll come to that bit in a moment and then I'll show you how it works I'm also conscious you asked a question that led into this which was why we started the demo which was a which was around the data versioning and so I just wanted I'll pause for a moment and show you that every time a run is captured that actually corresponds to a lightweight filesystem snapshot that's happening under the hood and we use a file system called ZFS which makes those snapshots instantaneous you can take them in less than a second and it also means that only the differences from one snapshot to the next need to be recorded and transmitted around which means for example that if there was some bulky data in here and then the next run added these test train and validation files only the exact blocks on disk that have changed from one room to the next need to be captured and that technology is really what makes data reproducibility feasible so yeah I kind of answered the question you asked 10 minutes ago okay and now this is maybe getting way deep into the weeds but is ZFS operate in user space like can you deploy dot science out to you know regular Linux machines and you have ZFS running within user space or you do you have to like run Linux on ZFS or something like that um so the FS is a kernel module and it runs as everyone knows this but ZFS has been available out of the box on a bun too since they think 1604 so if you're running a bun to you then you already have it it's already there you can just load it into the kernel and that science will also automatically install ZFS for you on on RedHat and and a couple of other distros docker for example the docker for Windows docker for Mac distro they all they will all also just work out of the box so you just mounting some ZFS filesystem it's you don't have to be running on a ZFS correct yeah from from the users perspective they don't have to do anything special if you're really interested in seeing what's actually going on in this container so this this Jupiter is running inside a docker container on the runner and you can see that this dot mesh filesystems dot mesh is the project that that we created initially which is a wrapper around ZFS basically we call it kind of get for data and that's exposed that ZFS filesystem as the home directory inside this juice' lab container add it okay cool great um so what we're gonna do next is I mean I actually just want to point out what happened here while we were talking which is that we trained this tensorflow model using the SGD optimizer and just for one epoch and then we tested it against the test set and it's got an accuracy score of 61% so it's not a very good roadsign classifier i wouldn't want to be sitting in an autonomous vehicle that's driving around with this thing on board um but never mind like let's just say I'm not a very good machine learning engineer I don't really know my way around tensorflow so I'm gonna have to ask someone for some help to make my model better um and just while I do that I'll just also mention what I just ran here is there's a command to to publish this this tensorflow model with this DST model command and what dear stock model says is hey dot science this is a model and this is where dot science starts to specialize to things like to the different machine learning frameworks everything we've done before this is generic like basically processes with input files and output files and keeping track of those but now we're saying hey dot science this is a tensor flow model and we actually pass in the tensor flow module as the first argument when dot science figures out from that exactly which version of tensor flow was used and then it builds an appropriate tensor flow serving docker container and then we do D s dot publish says hey we trained this tensorflow model so now let's go and look at the run which is this model run let's look at this run in a little bit more detail because this is starting to get interesting now and we can see we've have this piece of code that that ingested the sign names CSV file we also have the code which downloaded the these three data sets and those all went into this code which trained this model and the model the code which trained the model then created these four files the classes file and then the various tensorflow model files it also kept track of which parameters were used the epochs and the which optimizer and what accuracy score we got so um at this point I'm also going to it's also worth pointing out that we've got this Explorer tab for metric Explorer and so it's possible to say well this run at this time by this user got this accuracy score and obviously this this plot becomes more interesting when there's more than one model so at this point my hands up and say I don't I'm not very good at tensor flow I need to bring in an expert unfortunately one of my colleagues is an expert at this stuff and his name is Fred so I'm gonna add Fred as a collaborator um and then as soon as Fred has been added as a collaborator he can log into dot science and then he can start to take a look and so Fred comes in and and and he can take a look at the run and because he's an expert he can take a look and say ok well I can see that you trained this model on with with this dataset fine and and so on but he can also say oh look these parameters like that's not very good so you can leave a comment for me and you can say promises look dodgy let me have a go at improving it and I'll forget and make a PR for you and then of course from my perspective I can go in and I should be able to see the comments on this run and we're going to plug this all into slack so that every what everything's collaborative and people can get notifications of of these comments um but even without my permission because I've added Fred as a collaborator a Fred can do some asynchronous collaboration with me and so he's gone and created a fork and now his fork has got exactly the same set of runs in it that mine has but the interesting thing is that Fred can now kick off a separate jupiter instance that is it's it's separate from mine like we're not going to tread on each other's toes it's actually running on a separate VM in a separate container but we're both able to carry on doing work and make progress on improving the model in parallel now if we have more time in this demo I would show you how I can go and make some changes and then Fred can kind of merge from master in a sense to or merge the most of the changes from the upstream but because we don't have a great deal of time I'm just going to make do kind of the simple case where I haven't made any changes and Fred makes a proposal you made the merge that's within the context of the notebook exactly so we've done notebook dippin and merging as part of dot science so we'll be able to see that in a minute but yeah for the sake of argument like this is this this is Fred knowing what he's doing he's gonna change the optimizer from the SGD optimizer to the atom optimizer and he's gonna bump the number of epochs up to three so um and then going to leave this running because it'll take a little bit longer to firm for this model to train at three epochs and then I'm gonna go back to taluks world and then I'm gonna say well okay we've got a model it might not be the best model in the world but let's deploy it Lester play it to a Cuban ESI's cluster and then monitor the behavior of that model in production so the way I can do that is I can go to the models tab and because that model was declared in science using DSR model I'm now able to build that model into a doctor image and so I can click this button and notice that I didn't have to think [Music] versions of things I didn't have to think about ten Suvla serving or which port that containers going to run on or any any of that complexity I'm able to just click a button and das science will automatically build build a doctor image outs of the versioned model and that existed in my jupiter notebook now that's great if i'm the data scientists but if i'm the you know ml engineer deaf person I want might want a bit more control over what goes in that container is that something that is accessible to me um so that would be a great feature request we don't do that at the moment but yeah we currently use some some kind of pre baked doctor files here but yeah I think it would be totally reasonable to have the ability to configure the specific dr. feel that you use and the image that you create to add any additional dependencies so yeah that's something we may well add in the future okay cool so we've got this roadsigns model now and you've got the model status is complete you can now deploy it yeah so let's let's ship it let's put this thing into production so I'm gonna now going to choose a deployer and a deployer basically is a kubernetes cluster so you can attach multiple kubernetes clusters to your dot science account and by the way you can deploy science on Prem or in your own cloud VPC or use our SAS version so we have flexible deployment options and then you can attach however many kubernetes clusters you want to dot science by just running a single coop CTL command to to connect the cluster to science and so here we have a that science doesn't you know own abstract or kind of manage the cluster for me you're assuming that that's there and someone's taking care of it goodness using it that's correct and we can for clients that need support in standing up and operating a kubernetes cluster we can help them with that um it's also now easy to get a managed kubernetes cluster from all of the big cloud providers so we're kind of assuming that kubernetes is becoming a commodity basically and so I'm now going to deploy this road science demo to this to this cluster and I'm going to call it road signs which is not the most imaginative name but I can choose which kubernetes namespace to deploy it to I can scale the model up to more than one replica if I want to and then I can hit deploy mm-hmm and so I left it all at just the defaults and you can see that this is going to automatically deploy that model to a rest endpoint in that kubernetes cluster and it's also going to wrap that model up with what we call the model proxy and the model proxy is a sidecar that we created that people aren't familiar with what that is it's a another container that sits next to the main container in a kubernetes pod and it allows the container to it allows requests to go through the model proxy to the to the deployment and and in doing so we're keeping track of the requests and the responses and and that allows us to do what we call statistical monitoring so yeah this will deploy the kubernetes cluster and then we will be able to to monitor that cluster um so while that's deploying let's go back to the the model that Fred created and we can kind of interleave the different pieces of the demo here and we can see well okay by training it with the atom optimizer and three epochs we actually got a much better accuracy score so this model has now been has now been published and the and the version with three epochs and the atom optimizer is is is now listed in the dot science runs we can see this here as well so this is definitely an improvement I think in the terms of the accuracy score so we can now Fred can now propose this change back so Luke and so he can say let's make a pull request the pull request shows the diff going from the SGD optimizer up to the atom optimizer and going from one epoch to three epochs and Fred can go ahead and make that that pull request in the way that you're familiar with from github so Fred might say improving accuracy try training for more epochs and using atom optimizer is better in my experience and then Fred has gone ahead and opened that pull request um so if I go back to to the model for example then sorry to the project then I can go and look at my pull requests and I can see this improving accuracy the pull request is open and so I can say okay looks good I can go and review the diff I say okay well those are the things that have changed I'll I'll make that comment and and then I can accept the pull request so just before I do that I have to stop my local copy of Jupiter so that Jupiter and the the merge algorithm don't tread on each other's toes and and then I can go ahead and and merge this in oh do we uh I don't know if the models completed the deploy but can we get a quick look at that uh the monitoring side of things before we wrap up yeah totally um so let me just see okay so the model is up and running that error message is actually normal because you're meant to send post requests the model not get requests and so let's try monitoring the model the this is going to take us into agra fauna so what we do is we automatically create gravano dashboards for each of the models that are running in the kubernetes cluster and you can go in and you can edit the prom ql if you want to but you don't have to understand from ql in order to in order to use dot science and so you can see right now there are no requests go ahead the kind of monitoring we're talking about here is monitoring of the it looks like monitoring of the micro-service transaction rates and Layton sees as opposed to statistical monitoring of my decision service well actually it's it's both so let's send some requests of this model so I've got a little app here that is able to send roadsigns requests to models and we can start hitting up this this model that we're running in production and I told you it wasn't very good it thinks that this sixty sign is right of where the next intersection and and it thinks that this no okay got no entry right let's see it really likes right-of-way at the next intersection this is kind of a classic like not trained enough neuron they're here okay so it gets the stop sign right and get manages to do yield but yeah that's pretty bad I wouldn't want a model running in my autonomous vehicle that can't figure out what the speed limit is so so let's we can go and have a look at what what's actually going on here and what we can see here is I'll send it a few more requests just to make the the graph look a bit more interesting but yeah as we as we send requests into this model we're able to visualize monitor the statistical distribution of the different categories that the model is is is categorizing and so we can see here that even though we clicked on these buttons sort of equally we've got a spike in the number of right away at the next intersection predictions that are happening here and obviously little bit more clearly if i zoom in and and so using this statistical monitoring technique you can set up tolerances and you can set up using the Prometheus alert manager you can set up alerts to actually page a human page a data scientist if the model in production is is misbehaving or behaving outside of the normal bounds that you would expect and what an example would be that you might expect there to be a certain proportion of road signs that have print that a classified in production with these stop signs and if the number of stop signs that you're classifying in production drops below the expected threshold or even if it drops to zero then something changed either we deployed a bad model or something changed about the world and it turns out that these road sign classifiers that the model that we trained is incapable of predictive classifying stop stop signs in the snow for example like stranger things have happened with with machine learning models so so yeah that's the that's the statistical monitoring side of it and I guess just to sort of finish the demo here what we can see is that this this model that that fred contributed we can now go ahead and build that model into a doctor image and that you can see here that the is jump from 61% up to 86% and so what we should be able to do quite easily is deploy that model to the kubernetes cluster and then observe that in actual real world testing ie that this application that we that we built here that the model is able to is able to do a better job of classifying that sixty sign so I'm going to make a new deployment I just call it road signs too because I'm there was replaced existing deployment does that stand up your model behind an existing end point so you can update it without you know while it's running yes exactly exactly yeah and we're gonna soon add the ability to to have named deployments so you see at the moment every endpoint is just a random URL but we're gonna add the ability to name them and then you'll be able to have like production as a name for your field deployment and staging as a name for your deployment and that's going to be pretty useful anything fancy like canary deployments or things like that that's on the roadmap like once you get more than one model you definitely want to be able to test them against each other in in the real world so yeah Canaries and a/b is definitely on they're definitely on the roadmap okay um so yeah let's let's try out this other model this is the new one we created that Fred created oh and it's better it's not perfect at least it recognized it's a speed limit sign so I'm gonna call that a win and yeah unless there's any other questions I guess that's that's kind of the end of the demo awesome very cool very cool yeah I think we cover a lot of interesting ground here any kind of parting thoughts from your persuade I guess you know one of the that original question that led us into the demo was around kind of what enabled the Providence and the the things you're doing that on the data side and it sounds like that's the capabilities you built on top of ZFS would you say that that's kind of one of the core differentiators of what you're doing you know beyond the the user experience that you put together yeah totally um I think what we're finding is that number one companies need to get models into production and monitor them and if you can't get a model into production and then monitor it and tell what it's doing when it's running then you can't get business value out of AI and machine learning so that's kind of like level one of your hierarchy of needs as an AI team inside an enterprise but then once you once you've got models in production and especially once you start getting more models into production and new versions of models into production then if you haven't laid the groundwork for keeping track of your metrics and your hyper parameters let alone keeping track of which versions of data you're using as input data and if you don't have that reproducibility so that if a model is running nine months later and suddenly some stakeholder Flags that an issue with the model and you need to go back and and check for bias for example in the training data then then you're going to have a bad time so it's both the sort of enabling the essential value from AI with deployments and monitoring but then also keeping track of things improving collaboration and having that audit trail for the lineage the provenance of the model along with exact versions of which data went into each model and doing that all in a in a controlled collaborative and productive environment that's how I'd summarize it awesome awesome are you or anyone else on the team cube con next week yes my colleague Chris will be at cube con so yeah we don't have a booth at cube con unfortunately but if you tweet me /l master I suspect cube con will be passed by the time this is published but I'll be long and look out for for Chris yeah yeah you should definitely meet up with him and then I'm also going to be reinvent in December so and we will be exhibiting there so so yeah come and come and check us out awesome awesome well thanks so much for being a guinea pig and working with me on this experiment I think it was great for those of you out in the the viewing slash listening audience definitely reach out let me know what you think about this demo Wilson thanks so much sounds really appreciate it Cheers right