Deploying your ML Model with TorchServe

Key Takeaways

welcome to the pie torch summer hackathon 2020 I'm Brad Hinds and what partner engineer were the pie tours team if you're not sure what a partner engineer is it's my day job to make sure that developers like you get the absolute most they can out of pie torch and related tools in this video I'm going to show you how to deploy your machine learning model with torch serve a trained machine learning model alone does not make a full fledged application you'll need to integrate it into a larger system in this video I'll be using the term model serving to refer to this integration and subsequent use of the model recently the PI torch team released torch serve which is the preferred model serving solution for pi torch it's a performant and scalable tool for wrapping your PI torch model in an HTTP or HTTPS API in this short video I won't be going into the advanced features of torch serve instead I'll be focusing on the stuff you'll need for a hackathon namely how to stand up your first torch serve installation and start serving inferences from your model in this video I'll be showing you how to package your model and any supporting files it needs in a model archive and place it in the model store where torch sir can see it how to apply the default model handlers the torch cert provides for certain use cases and how to create a custom model handler if you need one for a new use case starting torch serve and configuring it for serving your model by a torch serves inference API examining a manipulating server state with the management API and where to go to find log files so you can see what's happening under the hood for this walkthrough I'm going to show you how to setup a simple image classifier and torch serve everything I'm about to show you is documented more thoroughly at PI torch org slash Docs and at github comm / pi torch slash serve first make sure you're in an environment with PI torch 1.5 or higher you'll also need open JDK 11 that's version 11 of the Java development kit next install torch sir and the torch serve model archiver you can use pip or anaconda for this I'll be using anaconda when installing with anaconda don't forget to use the PI torch channel that's - si PI torch on the command line now I'll show you some optional steps to optimize your model for use with torch serve they aren't strictly necessary but will often improve performance and memory footprint so I encourage you to try them and profile the results with your own model so first things first we'll import PI torch and torch vision and this is a good place to point out that you should be using pi torch 1.5 or higher for this example in the next cell we're going to create a PI torch model object now for the actual app I have a pre trained optimized model ready to go but for the optimization process I wanted to show you this on a custom model so that you could duplicate the process with your own models this model happens to contain some very common layer types A to D convolutional layer a to D batch norming layer and a rectified linear unit for activation and the forward function just strings those three operations together so now that we have our model how do we optimize it first thing let's get an instance of the model in my get model helper you'll notice that besides just instantiating the model I call m dot eval so eval turns off things in the model that you don't want on during inference time training holding layers like dropout automated gradient tracking all this training related stuff meets up CPU cycles and we don't need it for inference so we're gonna make sure the model is an eval mode the second thing we're going to do is some layer fusion fusing layers means taking multiple operations and combining them together into a single operation this improves performance in memory footprint now with diffuse modules method that I'm going to show you there are only certain combinations of layers that you can fuse together I'm going to refer you to the documentation for the latest information on that but here we're going to try to accuse together convolution bachelor-man Darrell you once modules are fused the next one we're going to do is quantize the model PI torch tensors default to using 32-bit floating-point numbers as their underlying type when we quantize we're going to change that to an 8-bit integer this will perform faster and reduce the models footprint both on disk and in marine the final thing we're going to do is save the model as torch script torch script is an optimized format for your model including both your computation graph and you're learning weights it's meant to be consumed by the PI torch just-in-time compiler or JIT so once it's exported we'll save now there are subtleties to layer fusion into quantizing your model that you'll want to be familiar with all of this is covered in the PI torch documentation for quantization which I encourage you to check out once you have your model ready you'll need to create a model archive and place it in the model store the model store is just a folder that holds your model archives a model archive is a single file bundle that includes your model and any supporting files you need for a natural language model that might be a vocabulary or an embedding or if you're deploying an on-court script model you'll also need to add the Python file that defines your model class so let's make the model store and we'll create a model archive for our model I'll be using my own model with the learning weight saved in a dot PT file and the model code in the dot py file I'm also going to pull down a single file from the torch sir github repo it's just a JSON file that has all the human readable labels for the categories my model is trained against let's take these three files and make a model archive out of them the important things I'm specifying on this command line are I specify the name of my model endpoint with the model name flag the version of the model tortured can simultaneously serve multiple versions of the same model for things like benchmarking and maybe testing with the serialized file flag we specify the file that contains our models learning weights the model file flag specifies the Python file that contains your models class definition the model handler flag lets you specify code that pre processes and post processes data for your model here I'm using the default image classifier model handler but if I were using a custom Handler I'd specify a path to a Python file with the model handling code for more details see the PI torch documentation and the github repo for torch servic github.com / PI torch / serve and finally the extra files flag lets you specify a list of files to include in your model archive here I've included the index that contains the human readable names of my categories so let's look at what we have and recap briefly we have created the model store folder gathered the files we need the model it's Python file and a supporting file with the human readable labels for our categories and finally we combine these files into a model archive file the final preparatory step is to move the model to the model store now we have everything we need to start torch serve and serve our model this is how we'll start towards surf the important arguments here are the start flag because we're starting torch serve the NCS flag stands for no configuration snapshots by default torch surf stays its state it will restore it the next time you restart it to avoid confusion for your first time using torch serve I've turned that feature off we specify the model store because that's where our model archives are and finally with the models flag we can specify a list of model archives that we'd like to serve you must specify the model archive file name the piece to the left of the equal sign is the name of the API endpoint and is optional so now I'll actually start torch serve and you can see a lot of log info scrolling by worker threads are being started models are being loaded into memory and those worker threads just for the sake of tidiness I'm going to flip over to another terminal window if you want to check the status torch serve and your endpoint you can use the management API this call tells us which models were serving we want to drill down on detail for a particular model we can specify it in the URL and now we can see that we have four separate worker threads all serving version 1.0 of my cv model you can also use the management API for adding and removing models and model versions and changing the number of worker threads assigned to a model all this is documented the docks at PI torch org and with example code at github comm / PI torch / serve so let's do some actual inference I happen to have an image file already and so let's see when model thinks this is for that we'll use the inference API on this command line we're going to specify our server and port by default torch serve uses port 8080 we're going to call the predictions endpoint and we're going to specify which model we want to serve the prediction and path to a file for input and sure enough my model thinks the picture of a cat is a picture of a cat so that's success the last thing I need to show you is how to stop torch serve you just call torch serve from the command line with the stop flag and that is torch serve in a nutshell there's a lot more to learn about torch serve multiple methods for configuration configurations snapshots for setup convenience and custom handlers for new use cases again I strongly recommend the documentation at PI torch org slash Docs and the examples at github comm / pi torch slash serve thank you for watching and for participating in a PI torch summer hackathon 2020