Lightning Talk: Orchestrating Machine Learning on Edge Devices...- Rishit Dagli & Shivay Lamba

Key Takeaways

uh hi everyone I know we it's the last session for today so it'll be pretty quick uh today's session is orchestrating machine learning on Neds with the help of pyot and web assembly I'm sh with mil and with me is Rashid from University of Toronto um so of course we'll be starting off with what is web assembly so web assembly is this binary instruction format that is specifically designed as a compilation Target for multiple languages whether you're talking about scripting languages or even objectoriented programming languages that can be compiled into this binary exec that can run as a platform agnostic uh where so basically you can run the same web simly binary on uh any platform it does not care whether you're running it on web or you're working on a host of different types of edge architectures it can run everywhere because it does not have to interface with the actual system architecture it just does not care which particular CPU or host OS you're actually running it on um of course uh webs itself cannot do anything on its own so it cannot uh directly interact with the file source ources or your networking resources for that we basically uh use wasi which is the web web assembly system interface so taking inspiration from how kernel kind of acts as the interface between your uh applications and the system itself for making CIS calls similarly wasi uh makes the CIS calls for your webs modules to be able to interact with file resources or networking resources um and of course we know that edge itself is fairly complicated right because of two main reasons first it's a res resource constraint so in comparison to your similar systems it's let's let's say 1 to 10% of the computer as compared to your standard uh competition and of course uh we have to deal with a number of different type of architectures x64 86 risk 5 to name a few which of course are pretty diverse in terms of their architecture so if you are having a use case where you want to deploy a machine learning model across a host of different type of device architectures you might have to make certain adjustments or optimizations uh to be able to run that model uh on such you know huge number of devices uh so that's where uh web assembly is really popular choice or a really great choice for being able to deploy workloads on edge with the help of web assembly uh and there are Prim three main reasons why uh you should consider webs as the choice first as I we kind of covered it's highly portable so it runs out of the box without having to make any specific uh you know integration or changes to be able to deploy it uh on a host of different types of devices and in comparison to the standard way in which you will run your python code or any other uh code on for that matter on an embedded device uh with the help of containers uh so in comparison to Containers the size of web assembly is pretty small and the startup time to actually invoke any web assembly function or for the module is a lot smaller uh so you are uh getting faster speeds in terms of the invocation and much smaller size so it makes it a very compelling choice to be able to run your work workloads and specifically since we are at a machine learning conference machine learning workloads across a host of different type of devices uh so that's why web is a fairly good chice and uh now we'll focus on uh arri so uh taking inspiration from kubernetes which is this orchestration platform that allows you to run uh ma different type of workloads very easily and you can manage them uh when it comes to like running cuberes on edge uh most of these small embedded devices do not have compute uh to be able to run cuberes natively so that is why we have this open source project called arri which essentially allows you to detect different type of leaf nodes and these could be just uh different type of heterogenous uh systems it could be like cameras or sensors and it allows you to kind of detect these different type of devices within a cuties cluster and then uh they kind of act as cuties resources and you can very easily just then run workloads on top of these devices so idea is that you could very easily just detect a lot of different type of uh these Leaf devices within a kubernetes cluster and then uh by detecting them and treating them as kubernetes resources you'll be able to run your machine learning workloads on these and here's a quick look at the project talki architecture so of course we'll start with our leave devices these could be a plethora of different type of heterogenous devices cameras sensors and of course uh if you're aware of the standard kubernetes uh you know architecture so we have the standard kuber control plan and within this we have primarily three major uh things that are specific to project arre so you have the arre controller arre agent and the custom broker so these all kind of together work uh in sync to be able to detect Leaf nodes for example the custom broker has this uh Discovery Handler that is working throughout the time to detect new type of edge devices and if it is able to detect an edge device then the arre agent will uh make this uh this particular device as a custom resource for your Cub cluster and you'll be able to very easily deploy your actual uh workload on this particular uh device and so on and so forth you'll continuously find more and more Edge devices throughout your entire system so over to rash to continue yeah so until now we talked a bit about uh maybe we want some way to integrate like all of these kubernetes capabilities within Edge devices to get all the benefits of Autos scaling orchestration and uh and we also talked about why wasum is a nice choice for Edge deployments so um so we want to talk about this a bit in the context of ML workflows and P Touch models so it is really helpful um to somehow uh integrate these how how do you do that we'll talk a bit about that but we would be able to use the software benefits we have with wasum so things like the sandbox uh having a thin sandbox without uh doing uh containerization and uh of being portable so it's not native uh you have the wasum runtime layer which makes it not native but uh but we'll soon see that it's actually pretty close to Native uh and and and since you have this wasum run time uh layer in between your modules or components or or each of the individual component instances also becomes really lightweight so you so we most certainly want a way to have these uh advances within our deployments as well so over here we'll talk about one of the examples we have a GitHub repository with quite a few code examples where we do models like Nerf which require fine tuning or training while inference and also standard models uh where we can just do inference uh and and we also have some other examples but right now let's take a look at uh this kind of example I I want to do novel view synthesis on my Edge device uh and and uh I'll try to implement mobile neres which is really popular uh and deploy that on my Edge device uh the setting we have is there are possibly multiple Edge devices and I want some way to do scaling over those Edge devices I want some way to do orchestration not just run them do all the other software aspects uh as well so this is until now what looks like our cluster we have the kubernetes control play which has all the standard things we have some sensor uh or the edge device uh represented by a camera here what our usern node has is this acree agent controller and the acre Discovery Handler the acri discovery Handler allows you to think of this as a cuberes resource so so now I can just think of this as I would a kubernetes node and just use all the benefits of the kubernetes right out of the box and what I also introduce is this wasum broker that allows me to uh schedule any wasum workloads uh using this shim ler because you don't want to uh use containers to do this so on a high level this is what it ends up looking like uh and and we run this wasum we schedule this wasum workload to run uh uh using uh the uh using the API server and commands to the control plane so in this case what we'll do is take a look at an example where I'll try to uh run uh run Nerfs and manage more importantly manage Nerfs in this fashion uh so so what I'll do is I'll assume we have all the mobile Nerf code uh compiled all the rendering uh and thought part compiled and uh what we'll try and do is uh think about how we would mainly write these aspects so but first of what I want uh want you to understand and is we have like this wasum module or wasum component that's being created from uh from inference code in any language and uh um we and and until now it's all portable but uh because we have like this wasm runtime layer in between but you could most certainly have something like aot artifacts and uh still get all the benefit of native and aot from a wasm module uh which would of course mean it sacrifices pability so actually let's take an example of uh uh let's take an example of what happens uh or I'll at least show a bit of it how much I'm able to show in this short talk uh we have a code repository where we not just do mobile Nerf but also instant NGP and other models uh so mainly what I want to show you is I I have some acre agents already deployed I also have my acre controller and I have these acre agents just like we talked about in the architecture now what happens is uh I I deploy all of these on uh on some control plane cluster and what my Discovery Handler looks like uh is uh in this case I just Implement a discovery Handler in Rust and then compile that down to web assembly to show that it's truly interoperable the IR is truly inter interoperable so so especially what I do in this discovery Handler is uh I I try to get this device and then uh through some protocol and then simply uh I I can simply start using all the kubernetes functions to do this and uh and and was though not native uh performs pretty well for mobile Nores uh and almost real time for these rendering results so uh so that was a short demo feel free to check out our code repository this was a very short talk so uh couldn't walk through the whole thing uh but uh that was about the talk thank you thank [Music] you yes so the question was are the Pyar operators themselves compiled to web assembly or do you link them somewhere so so there's an uh so there's ability to do module linking with uh wasi and wnn uh which we don't really talk about in this talk uh just for brevity but what you would do is uh within each of these component instances uh you also have something uh you have a module Linker for uh web assembly and uh and what that allows you to do is just run any of these other native uh or any of these other system binaries and use that kind of support for web assembly as well