TensorFlow Serving client examples

Key Takeaways

hi there welcome back to the video series of deploying production ml models with sensor flow serving my name is wei and i'm a developer advocate at google in our last episode we showed you how to start a tf serving model server and send post requests using the command line to occur while this is interesting we still need to learn how to send a request using different programming languages so in this episode we're going to discuss how to send rust and grpc requests to tf serving with python and c plus plus remember in our first episode we mentioned that your calendar app can send rest or grcc requests to tf serving and df7 will send back the corresponding responses it turns out that there are many clients in different languages and platforms you can use let's first take a look at a very common one using python to send rest requests to demonstrate this let's first download the risnet model from tensorflow hub and start tf serving with docker once the model server is up we can check out the model metadata in this link here we can see the input tensor name data type and shape it also shows the output tensor shape data type and etc it's usually a good practice to double check the metadata before you start sending requests to tensorflow serving since this is an image classification model we first download a test image file in our python code then we normalize the image sensor and dump it into json now we can send a poster request to the tf serving endpoint lastly we can get the prediction from the response that's it it's pretty straightforward right now let's move on to grpc grpc is a modern open source high performance remote procedure core framework that can run in any environment it can effectively connect the services in and across data centers with plugable support for load balancing tracing health checking and authentication please check out its home page to learn more about grpc grpc is open sourced by google and for our purpose it is an alternative way to send a prediction request to the tf7 backend let's see how we can leverage grpc in our python code we downloaded the image and normalized input sensor as before to send grpc requests we need to create a client stop which requires a channel and the server information in addition we specify our model name signature name and input tester in our prediction request now we use the stop to send the request to tensorflow serving once we get the response back we can extract the output tensors so this is how you would use python to send grpc requests but it is actually a little bit more involved than this we skipped this part to generate the client stop code because tf serving has done it for you in general to use grpc we need to define the remote service using dot product files then we can use grpc compiler to generate the client stop code which can then be integrated into your client this workflow applies to any programming language that grpc supports this is how our prediction service dot proto file looks like it has dependencies on a number of other product files including some in tensorflow repositories if you want to regenerate the client stop code yourself you can install grpc io tool package first and run the command to do so make sure you clone tensorflow repository as well since we depend on some products from tensorflow repository after the client stop code is generated we can import them and create a grpc request i highly recommend reading grpc documentation to learn more about generating grpc client stop if you are going beyond the python class bus lastly let's take a look at the c plus first c plus bus is actually very similar to python we specify our model name and signature name here then we read the image and use the client stop to send the request after we get the response we can extract the output sensor note that here we did not generate the client stop code either since tensorflow serving repository has done that for us so we just showed you how to call tensorflow serving backend using python and c plus plus what if you are not using python or c plus plus not to worry we have recently released a new set of codelabs covering web android flutter and ios foundants feel free to check them out so to summarize today we walk to you through how to send a prediction request to tensorflow serving backend with python and c plus bus you can find the complete code in the example folder of tensorflow serving repository in our next episode we are going to learn how to customize tensorflow serving please stay tuned thank you [Music] [Music]