TorchServe CPP Backend

Key Takeaways

foreign I'm a senior software engineer in AWS in this session I will introduce touch serve CPP backend today I will cover these topics first introduce our team and so why we view the CPP back end what's the change in architecture and then Deep dive the CPP backend internal give a demo and The Benchmark result touch serve is an open source project jointly developed by AWS and The Meta these are members for CPP back-end project why CPP backend there's a high production demand for lower model loading latency reducing GPU memory usage improving GPU utilization concurrency and even request embed touch serve in Edge device however it is difficult to support these requirements in Python backend due to Python language limitation so we decided to build the CPP backend to better serve thoughts of community now let's see what's the change in touch of architecture introduce the Byler CPP backend left aside a picture is touch server original architecture it is a German front end and the python candle worker process on the right side picture we can see backhand extended to support the C plus plus worker process Java front end is not only able to connect with the python worker process but also able to connect with the CPP worker process let's deep dive CPP backend internal as same as python backend CPP backend is divided into multiple layers top layer is Socket to communicate with Java front-end second layer is service management which is responsible for request response encoding decoding and dispatching the requirements third layer is common backend API to support different machine learning plugin by default touch serve provide touchscrip.back end the bottom layer is machine learning platform Handler API for customer model loading pre-processed processing and inference the bottom two layers are highly related to types of users as same as python backend I will explain them one by one the common backhand API layer defines three virtual functions for different machine learning platform to plugging function initialize is from machine learning platform initialization function load model internal is used to implement the model loading in different machine learning platform function predict is used to implement the prediction in different machine learning platform touchscripto backend is the default CPP backend as same as the python backend Handler CPP backend has default base Handler implementation but the users can override it function initialize it use is used for model initialization function load model is used for model loading function pre-post the process is used for model pre and post processing function inference is used to call model to wrong prediction CPP backend maintains the same behave as python backend to allow users to build a model map file by using model archival command for example users is able to use touchscript the base Handler to view the model map file user is also able to build a customer Handler Dynamic library and then wrap it in model Mafia by using model archival command finally users are able to load model and the wrong prediction by using same rest API or grpc API as python backend let's take a look at a demo this is the CPP back and the user menu page to save time already installer code this is a CPP backend common API links this is a touchscript the base Handler API links let's see two use case to build a model my file first I'm you I'm using touchscript the base Handler to build a model my file and then move the model map file into the model store second I'm going to using customer Handler Dynamic library and wrap it into model map file foreign into model store let's start a torch serve we can see the two models are loaded and then we run the model inference this is a model text classification by using script tokenizer Benchmark result the model size is about a 1.9 gigabytes the first picture shows model loading latency is reduced about 25 percentage the P90 and the P99 inference latency also reduced when the batch size is 1 the P99 latency is reduced about the 10 percentage wisdom exercise increasing the gap between CPP and depends on backend is reduced these are the GitHub links for touch soap and CPP backend please try and give us feedback thank you