Getting Started with NVIDIA Triton Inference Server

Key Takeaways

[Music] crater infant server is a robust feature-rich inferencing solution designed to deliver fast and scalable model deployment the breadth of features available can lead to a natural question for new users where do we begin let's start by looking at how to deploy models first we need to build a mod repository which is the organizational hub for the triton entrance server this repository houses all the models configuration files and any other resources needed to serve the models second step is to launch the server this may take a moment depending upon the number of models being loaded once loaded users can see which models are ready for inference and what ports can be used to serve the requests lastly we can query the trident inference server via http or grpc request in this example we are using the densenet model to classify our coffee mug let's peel back some of the abstractions and have a look at what happened under the hood to make this query possible an inference request can be sent to triton via the grpc http or capi in this case we will build a python client and use the http api as inference requests come into triton by default they are served sequentially but they can be backed together to provide better performance each of the models has its own scheduler which maintains a queue for that model's influence the model themselves can be dynamically loaded from cloud storage or local file systems without restarting the server users can also specify the model loading policies to tailor to the use case upon queuing the schedule will send the request to the appropriate framework backends to perform the actual computation the output tensors are then sent back to the client this was a quick rundown about the flow of an inference request through triton now with the general picture in mind it is highly recommended to check out features like dynamic patching concurrent model execution inference accelerators model ensembles and more all of which can be used with just a few changes to the configuration file we also encourage you to try out tools like the model analyzer and the model navigator which help optimize model deployment you can find the links to the documentation and other helpful information below this video happy inferencing