Hosting Models at Scale

Key Takeaways

thank you uh so we have one more talk um you may have heard through some of the previous talks uh speakers referring to metaflow hosting which is not something in open source but you know it's similar to what Bento ml would do that has been described as well uh so shashan will from the Netflix metaflow team as well will describe what metaflow hosting is and how we use it thank you um hi so I'm shashan and we have talked a lot about metaflow hosting in previous talks like aliki stock mentioned how they use a leverage metaflow hosting in order to do predictions for their DSC projects and then we also had the Amber team that is the media ml team talking about how they leverage async hosting for async endpoints for computing Amber features and we are just going to talk about the infrastructure behind metaflow hosting in the stock so the first question is why do we need metaflow hosting and yeah so when you think about machine learning at Netflix the first thought that comes to your mind is recommendations or predicting or finding out which movie a user would want to watch and there is indeed a lot of research or machine learning being used to do the same but this lies directly on the consumer side of inference metaflow hosting on the other hand is used for internal use cases it's based on a python hosting model that is flask framework so it's definitely not meant for Consumer Scale Models the places where metaflow hosting is useful is for internal UI tools like identifying the quality of network or performing machine translation for subtitles stuff like that and this is where metaflow hosting comes into play the other reason why we need metaflow hosting is that there's a lot of difference in what data scientists expect uh expect the code to be and what INF teams need to provide them so there's this nice cartoon or comic about every cloud architecture in Big Tex you have like these cool databases and then the mismanaged services or like the unmanageable services and there's also your good old data leaks which sometimes leak and lead into a data swamp but the point is that as a data scientist you don't want to interact with all of this all you care about is train a machine learning model write some simple code and then when actually you deploy a model you want to do some Quest tracing and be able to debug your model easily on the other hand the infra teams care about some aspects mentioned here such as load balancing scaling and an infr team might also care about like how do we bake the docker images like how do we configure the Gateway so that the end points are rooted correctly and what's the underlying infra for the infr team within Netflix such as are we using kubernetes or are we using ec2 S3 and so on so metaflow hosting kind of tries to reduce this gap between the infr teams and data scientists by providing a simple mechanism for data scientists to De to Define Services which they can then use to deploy the machine learning models or any AR arbitrary rest end points so what is metaflow hosting before we go with that like um I assuming that a lot of folks are familiar with metaflow but maybe some aren't so at a high level a metaflow is just a simple class which represents a dag each function or step is what we call it represents a node in this dag and you can use this dag to execute code in containers or you could also execute it locally and metaflow internally as well as an open source provides a lot of features like you can schedule your flows across or orchestrators like in OSS you can schedule flows in arbo on the other hand we have some internal orchestrators like Maestro within Netflix and then for each of these steps you can also specify the resources at Titus is just uh orchestration platform we have within Netflix which is based on Amazon ec2 as well so the other important thing to note is that each step in your metaflow flow can generate these things known as artifacts and these artifacts can then be used in your deployed models so the way a deployed model can make use of artifact is shown here this is the entirety of code that a data scientist needs to write in order to deploy the service across multiple instances have load balancing as well as proper request tracing and observability metrics so you have the example web service defined which inherits a particular class and you get specify the resources needed as well as the autoscaling params like minimum or maximum instances then you can use this function called inet app to initialize your application with whatever relevant models you need or you could also specify if you need some particular package finally we have this at endpoint decorator where you specify the name of your endpoint followed by accessing the artifact so this artifact could be a machine learning model that you just train via a metaflow flow like shown in the previous slide and in this function you have this request dictionary wherein you can get the Json body for your HTTP request and then run your model on it to get a response and deploying the web service is just as easy as running this command wherein you specify the flow that you want your service to be associated with and the definition of your service metaflow hosting provides a lot of features like this is what I mean by simple definition and simple deployment there's also the notion of dependency management where you can specify the environment name and that basically installs the relevant pip packages or cond packages needed for your service we support Auto scaling which is like scaling according to the number of requests there's versioning traceability and support for ASN requests so this is what a deployed model looks like you have the model deployed across multiple instances at the top when you make a single request to an endpoint you can trace the request via elastic search indexes and we also support tracing via Edgar or Zipkin you have logging via radar wherein you can see how your service is performing with respect to the status codes and finally you have the good old metrics which you can use to find out the status of your service whether it's up or not and this is actually used by US to do the Autos scaling as well like if we see a lot of requests coming in then we have a job or a service which can autoscale your service so moving on to implementation details there are multiple ways of implementing a metaflow hosting application internally but there are pros and quants for each of them so we considered DNS redirection as one of the options but we want to look at all of these options based on these four criterias consistency simply means that when you deploy a new version of your service the redirection should be able to properly redirect to that latest version of service and this is an issue with DNS redirect because it's based on a caching mechanism and if you just deployed a new service the cach would not be updated immediately and if the user makes a request to the latest version of endpoint they might still be rooting the request to an older version the other criteria we are interested in is low maintenance and that's particularly important for us because we are a small team and we do not want to maintain the infrastructure for a lot of services so while proxy server and open fast both support uh lot of these features open fast was the best for us because it's low maintenance because there's another team within Netflix maintaining it for us and then there's some other features like logging tracing and load balancing which is supported by both of these and for those unfamiliar with open FAS open FAS is an open source framework for like serverless functions kind of functions like AWS Lambda but and open source this can be based on any back end so you could have kubernetes as a back end or in our case we use Titus as a backend Titus is also another orchestration platform within Netflix so yeah we chose open fast for our metaflow hosting implementation the metaflow hosting implementation consists of two main aspects one is the control plane and one is the metaflow client or the container so the control plane itself is written and go it's highly performant and it is responsible for creating the instances for a new service and control plane is also the service which Roots a query from the users's machine to a particular microservice or endpoint the control plane also does other fancy stuff like generating a Swagger for the user and it's also responsible for maintaining state of the World by maintaining state of the world we mean that it needs to know which versions exist right now which versions can be deployed or undeployed and which version corresponds to the latest production version on the other hand we have metaflow client and containers which is the the code that the data scientists actually interact with so the metaflow client essentially downloads and packages the users code within a Titus instance and then it fetches the relevant artifacts like I mentioned earlier we then do some user directed initialization and dependency management that is download the relevant packages and finally we serve the HTTP endpoints using flask and fatdog and F Watchdog is also based on go in order for it to be a simple server that is highly performant so going over this quickly like we have open fast Gateway which we fondly call Barkley internally and based on this Gateway we create another service called the control plane that I mentioned in the previous slide and then this control plane interacts with a RDS instance which is a database which stores the state for all our hosting functions and let's go over the deployment path so when a user deploys a new service we first go to the open fast Gateway which then makes a call to the control plane the control plane then fetches the relevant information from the database to find out whether the service has already been deployed and we're deploying a new service or whether we want to create a completely new service after that the control plane returns back to the open fast Gateway which interacts with our orchestration platform form to actually start the services needed on the other hand we have yeah so this is what finally deploys the hosted microservice and you can have multiple instances of the same now going over the query path if read his strings wants to query a particular endpoint within our hosted application the query first goes to the Gateway which again goes to the control plane which uses the DB to root it to the appropriate end point goes back back to the Gateway again and then actually makes the call to the micros service or the hosted front end and finally we get the response back so that's metaflow hosting at uh at a high level and then we continue developing metaflow hosting internally like one of the features that was added recently was async hosting wherein you can have long running jobs so in general metaflow hosting supports a request for a period of 20 minutes but with async hosting we support jobs that can run for as long as 12 hours this is particularly useful for the media ml team because they run inference on movies which are pretty large or long and take a long time to perform to execute and the reason for the 12-hour time limit is because of the sqsq which is what we used in our implementation and this is an example of of how you can use callink to get the response from an e sync request but you can also have a simple call back function to do whatever you want and one of the latest features that we integrated into the metaflow hosting pipeline is adding support for a graphql backend uh why do you want support for graphql backend at a big company like Netflix you can have multiple uh microservices owned by different teams as a client UI engineer who defines or creates front ends you don't want to make calls to like 10 different microservices in order to get the response for making a UI page having a Federated graphql Edge allows you to make one single query and then this Federated Gateway allows you to root the queries to the appropriate microservices get back the responses from each one of them it collates them and then returns a unified response back to the client UI engineer so in order to support a Federated Gateway we recently added support for defining a graph Q backend in metaflow and it's pretty seamless for a user to add this there is literally no change in their service code like the service code Remains the Same except within their endpoint they can Define the name for the graph C endpoint as well as the input type and and output type graphql is a typed language unlike python so you need to Define the types like here and then we do all the associated stuff like deploying it to the Federated Edge and that's it any questions thank you