TensorFlow Extended (TFX) Post-training Workflow (TF Dev Summit '19)

Key Takeaways

[Music] hi my name is Christina and I'm a software engineer on the Google brain team I'm here today to tell you about some tools that my team and I have built to help make the end and lifecycle of the machine learning pipeline easier I'm going to start by talking about model analysis and validation these are two different components in tf-x but they are very similar in how they're actually executed the main difference is how you as an end-user will use them I'm going to start by talking about the evaluator so why is model evaluation important well for one thing we have gathered data we've cleaned that data we've trained a model but we really want to make sure that that model works and so model evaluation can help you assess the overall quality of your model you also may want to analyze how your model is performing on specific slices of the data so in this case with the Chicago Taxi example that Clements started us off with why are my tip predictions sometimes wrong slicing the data and looking at where you're doing poorly can be a real benefit because it identifies some low-hanging fruit where you can get gains in accuracy by adding more data or making some other changes to make some of these segments improve you also want to track your performance over time you're going to be continuously training models and updating them with fresh data so that your models don't get stale and you want to make sure that your metrics are improving over time and not regressing and model evaluation can help you with all of this the component of tf-x that supports this is called the evaluator and it is based on a library called tensor flow model analysis from the pipeline perspective you have inputs which is your eval set that was generated by your example you have the trainer out putting a saved model you also need to specify the splits in your data that you find most interesting so that the evaluator can pre-compute metrics for these slices of data your data then goes into the evaluator and a process is run to generate metrics for the overall slice and these slices that you have specified the output of the evaluator is evaluation metrics this is a structured data format that has your data the splits you've specified and the metrics that correspond to each one of these splits the tender flow model analysis library also has a visualization tool that allows you to load up these metrics and dig around in your data in a user-friendly way so going back to our Chicago taxi example you can see how the model evaluator can help you look at your top-line objective how well can you predict trips that result in large tips the TF maa visualization shows the overall slice of data here the numbers are probably small but accuracy is 94.7% that's pretty good you'd get an A for that but maybe you want to say 95% 95% accuracy is a lot better number than 94 94 point seven so maybe you want to bump that up a bit so then you can dig into wider tip predictions or sometimes wrong we have sliced the data here by the hour of day that the trip starts on and we've sorted by poor performance when I look at this data I see that trips that start at like 2:00 3:00 a.m. we're performing quite poorly in these times because of the statistics generation tool that Clemons talked about I do know that the data is sparse here but if I didn't know that perhaps I would think maybe there's something that people that get taxis at 2:00 or 3:00 in the morning might have in common that causes erratic tipping behavior someone smarter than me is gonna have to figure that one out you also want to know if you can get better at predicting trips over time so you are continuously training these models for new data and you're hoping that you get better so the tensorflow model analysis tool that powers the evaluator and tf-x can show you the trends of your metrics over time and so here you see three different models and the performance over each with accuracy and AUC now I'm going to move on to talking about the model validator component with the evaluator you were an active user you generated the metrics you loaded them up in the UI you dug around in your data you looked for issues that you could fix to improve your model but eventually you're going to iterate your data is going to get better your model is going to improve and you're going to be ready to launch you're also going to have a pipeline continuously feeding new data into this model and every time you generate a new model with new data you don't want to have to do a manual process of pushing this to a server somewhere the model validator component of tf-x acts as a gate that keeps you from pushing bad versions of your model while allowing you to automate pushing of quality models so why model validation is important we really want to avoid pushing models with degraded quality specifically in an automated fashion if you train a model with new data and the performance drops but say it increases in certain segments of the data that you really care about maybe you make the judgment call that this is an improvement overall so we'll launch it but you don't want to do this automatically you want to have you want to have some say before you do this so this acts as your gatekeeper you also want to avoid breaking downstream components if your models suddenly started outputting something that your server binary couldn't handle you'd want to know that also before you push the tf-x component that supports this is called the model validator it takes very similar inputs and outputs to the model evaluator and the libraries that compute the metrics are pretty much the same underneath the hood however instead of one model you provide to the new model that you're trying to evaluate and the last good evaluated model it then runs on your eval split data and compares the metrics on the same data between the two models if your metrics have stayed the same or improved then you go ahead and bless the model if the metrics that you care about have degraded you will not bless the model get some information about which metrics failed so that you can do some further analysis the outcome of this is a validation outcome it just says blessed if everything went right another thing to note about the model validator is that it allows you to do next day eval of your previously pushed model so maybe the last model that you blessed it was trained with old data with a model validator it evaluates it on the new data and finally I'm going to talk about the pusher the pusher is probably the simplest component in the entirety effects pipeline but it does serve quite an amusing that you got from the model validator and then the output is if you passed your validation then the pusher will copy your saved model into a file system destination that you've specified and now you're ready to serve your model and make it useful to the world at large I'm going to talk about model deployment next so this is where we are we have a trained save model a saved model is a universal serialization format for tensorflow models it contains your graph you learned variable weights your assets like embeddings and vocabs but to you this is just an implementation detail where you really want to be is you have an API you have a server that you can query to get answers in real time or provide those answers to your users we provide several deployment options and many of them are going to be discussed at other talks in this session tensorflow Jas is optimized for serving in the browser or on nodejs tensorflow Lite is optimized for mobile devices we already heard a talk about how Google assistant is using tensorflow light to support model to support model inference on their Google home devices tensorflow hub is something new and andrey is going to come on in about five minutes and tell you about that so I'm not gonna step on his toes I plan to talk about 10 to flow serving so if you want to put up a REST API that serves answers for your model you would want to use tensorflow serving and why would you want to use this for one thing tensorflow serving has a lot of flexibility it supports multi-tenancy you can run multiple models on a single server instance you can also find multiple versions of the same model this can be really useful when you're trying to canary a new model say you have a tried and tested version of your model you've created a new one it's passed your evaluator it's passed your validation but you still want to do some a be testing with real users before you completely switchover tensorflow serving supports this we also support optimization with GPU and tensor RT and you can expose a gr PC or a REST API tensorflow serving is also optimized for high performance it provides low latency request batching so that you can optimize your throughput while still respecting latency requirements and traffic isolation so if you are serving multiple models on a single server a traffic spike in one of those models won't affect the serving of the other and finally tend to flow serving is production ready this is what we use to serve many of our models inside of Google we've served millions of QPS with it you can scale in minutes particularly if you use the docker image and scale up on kubernetes and we support dynamic version refresh so you can specify a version refresh policy to either take the latest version of your model or you can pin to a specific version this can be really useful for roll backs if you find a problem with the latest version after you've already pushed I'm going to go into a little bit more detail about how you might deploy a REST API for your model we have two different options for doing this presented here the first the top command is using docker which we really recommend it requires a little bit of ramped up at the beginning but you will really save time in the long run by not having to manage your environment and not having to manage your own dependencies you can also run locally on your own host but then you do have to do all of that stuff long term I'm going to go into a little bit more detail on the docker run command so you start with docker run you choose a port that you want to bind your API to you provide the path to the saved model that was generated by your trainer hopefully it was a it was pushed by the pusher you provide the model name and you tell docker to run the tenta flow serving binary another advantage of using docker is that you can easily enable hardware acceleration if you're running on a host with a GPU and the Nvidia docker image installed you can have you could modify this command line by a few tokens and then be running on accelerated hardware if you need even further optimization we now support optimizing your model for serving using tensor RT tensor RT is a platform for Nvidia for optimized deep learning inference your chicago tax the example that we've been using here probably wouldn't benefit from this but if you had say an image recognition model a ResNet you could really get some performance boosting and cost savings by using tensor RT we provide a command line that allows you to convert this saved model into a tensor RT optimized model so then again a very simple change to that original command line and you're running on accelerated GPU hardware with tensor RT optimization so to put it all together again we introduced tensorflow extended or tf-x we showed you how the different components that tf-x consists of can work together to help you manage the end to end lifecycle of your machine learning pipeline first you have your data and we have tools to help you make sense of that and process it and prepare it for training we then support training your model and after you train your model we provide tools that allow you to make sense of what you're seeing of what your models doing and to make improvements also to make sure that you don't regress then we have the pusher that allows you to push to various deployment options and make your model available to serve users in the real world to get started with 10 to flow extended please visit us on github there's also more documentation at tensor flow org slash tf-x and some of my teammates are running a workshop tomorrow and they'd love to see you there you don't need to bring the laptop we have machines that are set up and ready to go and you can get some hands-on experience using tensorflow extended [Music]