Introduction to ONNX Runtime

Key Takeaways

hey everyone I'm Peter from vide Tech marketing team and today I'll be walking you through an introduction to Onyx Onyx runtime DML or direct ML and Onyx runtime gen uh what each of these Frameworks or libraries are how they interact with each other and how they're distinct from each other uh I'll expect no prior knowledge in any of these Concepts um but some basic familiarity with llms or AI models in general starting off at the top what is Onyx Onyx is an acronym that stands for open neural network exchange and the idea behind the project is that when you are creating a model as a researcher there are a lot of different Frameworks where you can Implement and train a model on the screen right now is sklearn tensor flow and torch uh and you can choose whichever one fits your use case whichever one uh supports the kind of model features that you want and once you train it that's all well and good but then you have to deploy it on a certain Hardware you have to send it out to be inferenced upon either on a CPU on a GPU on a mobile device on an npu all of these different Hardware options and you don't want to be thinking about that while you're training the model uh you want that to be I have a model that is trained let me now think about deployment and so Onyx is an effort to decouple the training and the deployment options for people creating models and given those considerations Onyx is a standard to represent any machine learning model that is focused on inference with the idea being that you can train in any framework you want and then take advantage of a converter tool to turn your model into an onyx model and then move on to the deployment process a very basic look at what the standard looks like is a model in the form of an execution graph where each node represents a basic operation being done either a matrix multiply and add to subtract uh and the vertices in this graph represent the dependency structure of each operation now that we have an idea of what Onyx is what an onyx model looks like and why the standard exists uh we're going to go into what is Onyx runtime Onyx runtime is a library that orchestrates the execution of an onyx graph and actually connects these operations uh these nodes and vertices to actual executable code in the middle here is an example of what an onyx model might look like and Onyx runtime as a framework will take this execution graph and convert it into a series of operations on the right where we'll say the operation that needs to be done the arguments to that operation where that output is going to be held and it's going to work its way through all of these operations until it reaches the very end where it takes the model output a core concept to onx runtime is the idea of an execution provider and what is an execution provider um well once Onyx runtime has generated a table like this like we showed in the last slide um it has orchestrated the graph into a series of operations it says all right I know what I want to execute I have the inputs that I need to throw into that operation and I know exactly where I'm storing the output but at this point Onyx runtime doesn't actually have any AB ility to execute these operations uh it knows that it needs to execute a matrix multiplication but it doesn't know how um and that's where the execution provider comes in um it literally provides the execution details for all of these Onyx operators that are found in the Onyx model you're trying to run in Onyx runtime and the general flow of this is that uh Onyx runtime will go down this table and say all right I have the operation that I want to run I have the inputs that I want to run it with I am going to kick that down to the execution provider that I'm currently running and I'm going to execute the code that the execution provider holds for this operation I'm going to take the result kick it back up to Onyx runtime store that in the output and then rinse and repeat for every single row that we have in this uh operation table uh until we're done there are a lot of different execution providers supported by Onyx runtime and they all differ with the underlying code that comprises the definitions of the Onyx operators where that code is executed and how optimal it is these all differ EP to EP I'm going to focus today on the direct ml or DML EP uh because it's a super flexible option uh very performant and it's a active for running code on your Nvidia gpus or any hardware that supports directl now that we have an idea of what Onyx is what are onyx models what is Onyx runtime and what are execution providers uh we can dive into the last layer here which is what is Onyx runtime gen and the idea behind the project is that when working with generative models uh there's a lot more outside of core model inference that needs to be done to provide an endtoend experience to a user or developer and as an example I have a little diagram here of what it might look like to inference llama 3 or any generic llm uh you have some input what is the capital of France you're going to use Onyx runtime like we've talked about to inference the Llama 3 model and the output that you're going to get is actually a probability distribution over a lot of tokens in a vocabulary at this point you have inference llama 3 using Onyx runtime but now you have to write wrapper code to sample the token that you want maybe cross out tokens that you don't want if you have a vocabulary mask and you want to get rid of these you want to remove them from the probability distribution um you want to sample a token from here append that to the input that you have so far and handle any caching of values that needs to take place and then rinse and repeat where you pass the input through llama 3 again get another probability distribution sample and then uh do that in a loop until you hit some terminal condition um and the idea of Onyx runtime gen is that it uses Onyx runtime in this way to inference the llm at its core but all of this wrapper code is handled for you through this Library so ultimately this sent represents the the relationship between Onyx runtime gen Onyx and Onyx runtime and that being that Onyx runtime gen as a library will manage repeatedly executing a generative Onyx model using Onyx runtime to provide an endtoend flow to you as a user and this right here is a screenshot from the official Microsoft GitHub page uh it outlines some of the specific steps that I'm talking about with this decoding loop with logic processing search sampling cache management it's all things that are handled by this Library another way to put it is this little diagram Onyx runtime geni is a generative framework that handles the looping execution of Onyx runtime which utilizes an execution provider to orchestrate the inferencing of Onyx models hopefully all of these Concepts and how they interact with each other and are different from each other makes sense at this point we're going to dive into how to actually accomplish this in code