ExecuTorch 1.0: General Availability Status for Mobile and Embedded...- Mergen Nachin & Cemal Bilgin

Key Takeaways

Hello everyone. Very excited to be here with all of you. Um my name is Bill Gin. I'm the engineering manager for PyTorch on device. And I have Morgan with me who is the tech lead on um on PyTorch on device ML. And today we've got some exciting news for uh for everyone. And uh we'll be covering the ExecTorch 1.0 release. Uh this is our general availability release that we've been building together with you folks over the past few years. And um Exeutor is our is PyTorch's ondevice uh AI um solution. Um we have been building Executive uh to um enhance privacy right by making your data stay uh within your device. For example, you can do translations for WhatsApp. Um right. Um it improves the uh latency by making sure that the data is not um requiring to uh do that round trip to the servers. Um improves the cost um saves the cost by uh running these ML workloads um at the edge uh rather than those expensive GPUs. Um it gives you uh remote access uh without requiring um networking and uh um a few other reasons for doing on device ML um is uh you know especially on the LM side on genai um the models are getting smaller and they're getting smarter. What was um what a larger model used to be able to do a couple of years ago are now available with much smaller models on the edge. Um the uh those are some of the benefits of one device ML but there are some fundamental challenges as well that we had to um we have to solve for battery life and power is one of these. Um these tiny devices um it's very easy to drain the battery. Um these edge devices also have a lot of memory constraints memory bandwidth um thermal constraints. It's very easy for your classes to heat up when you're running these workloads on on the edge and the hardware heterogenity, right? We have um multiple different chipsets um and um the need to optimize for each of these platforms is um is a big challenge. Um and um to solve for those challenges um we um ground up I you know built Exeutor and um with um within with with Executtor we were able to stay within the um same uh PyTorch ecosystem uh from authoring all the way to deployment. Um you start with your PyTorch program and um with uh torch export we're able to do full graph uh capture and with progressive lowering um we're able to get that um PyTorch model into the edge devices and uh without requiring any conversions or rewriting um we're able to solve uh for this problem. Um couple of examples on showcasing some of the simplicity um some code snippets um ahead of time. It's as simple as you know um instantiating your model and exporting the model um with tortlet export and then uh you know later stages you can send your program uh program to edge um when this particular case we are using um the x&m pack partitioner u to to you know utilize the um CPU accelerators that we have and uh finally you're writing your program into uh the flat buffer. Um on the runtime side um with the given model uh model file uh with a four lines with four lines you're able to up your model file and you know create a example uh tensor and um simple module forward. Um you're able to get your inference. Um we're giving some examples on the C++ side. Um we have the same um simple APIs on the Swift side u for iOS and we also um have the um similar simplicity on the Android side with Cotlin as well. some design principles that I'd like to um emphasize. Um hopefully most of these are uh not news. Um we wanted to make sure that uh Exector is portable, right? Supporting a wide variety of age platforms. Um from powerful mobile chipsets to embedded systems um and performance, we wanted to make sure that the runtime is lightweight. Um and uh um with strong partnerships with the um hardware partners um we've optimized the the performance as well. Um productivity staying within the PyTorch tool chain u making sure that you are able to go all the way from um authoring to deployment uh without necessarily needing these um sometimes awkward conversions um and uh debugging experiences. Um these were some of the design principles. Obviously we wanted to do this in open source with the community. Um uh that was the driving factor for us to um you know release execut two years ago with an MVP. um wanted to build it together with you folks and uh um modularity we're providing uh well- definfined um workflows for PyTorch programs from capture to uh transformations and execution um with um out ofbox components. Um in terms of the timeline just giving a little bit more history um started with MVP this was mostly to gather feedback and again uh build executive this stack together with all of you um shortly after that um we've added llama support and uh the our alpha release um this is when you know LLM um support and executed picking up and we improved our SDK quite a bit. uh there were a lot of delegate um enhancements and last year um during this conference we provided some strong performance uh numbers on LLMs um and we gave some early adoption stories um and uh we also um provided some API stability and uh today finally after um all of that hard work um we are going to be uh you know, going through some of our um ecosystem integrations. We're going to talk a little bit about adoption. We have strong adoption. We're going to talk uh we've we've improved the backend coverage and reliability. And uh we'll give some of some of those examples. Um without further ado, again, um as of today, we are generally available and uh with multiple success stories. I'll start with some of the internal adoption. Um, Exeutor is integrated into um all of Meta um apps. It is integrated into Facebook, WhatsApp, Instagram, and Messenger. It's powering multiple different use cases um serving billions of inferences for billions of users. Um it's required a ton of reliability and hardening work. Um and uh um it not only powers the uh family of apps um it also powers our RL headsets and smart glasses. There are multiple use cases that we are running um on the Oculus headset and Rayban um smart glasses. Um for external success stories we've partnered with liquid AI, Private Mind, Nimble Edge um and um multiple models are are getting served um by these external partners. There are a number of ecosystem integrations um with software mention um uh they integrated executtor uh and are built are have built react native exeutor serving um multiple users of theirs. Um we partnered with hugging face um to um increase the model coverage through optimum um and um torch.io uh for um interoper operability on the quantization side, onslaught AI, ultraics and media flare for um ondevice training and diga for a number of use cases that they're supporting and these ecosystem integrations and adoptions is only increasing and we expect a lot more. Um um [sighs] another area since um that we have improved since the um beta release is uh the backend coverage. Um we have XNM pack with um ARM CYAI for CPU acceleration. Um Apple CoreML u for Apple silicon and Qualcomm AI for the Qualcomm hexagon MPU. um ARM ethosu for MPUs and Vulcan GPU. These are all u production ready um um serving um uh multiple use cases uh for for meta. Um there are a number of other backends that we have added over the past 6 months to a year. ARGF, NXP for Neutron MPUs, Samsung um Einos for ENOS and MPUs, Open Veno uh from Intel has been added um and um Cadence DSP uh MediaTek NPU, Apple um MPS will continue to harden these uh delegates and make sure that they are um serving uh millions of users as well. Um a few more um features before I give it to Morgan um that we have uh developed for the general availability. Um as I mentioned one of the key areas where we um um worked hard uh for this release was uh reliability and hardening. Um serving billions of uh users and multiple use cases for different variety of hardware is no easy task. Um we Exeutor is now um at meta scale able to uh scale and provide that reliability. We've got strong coverage and performance guarantees thanks to uh the Qualcomm team over here on the Qualcomm engine back end especially on LMS. Some of those numbers are going to be available on our uh website. Uh with the ARM on the ARM side, Clyde SME SME2 enabled and um you know multiple use cases run efficiently on um low power ARM ethosu. Um, and we've also added Windows support um, and improved the UX um, for build and swift PM and Maven um, builds. Um, we've added program data separation um, and for example, you can run Laura um, which was a long requested uh, feature from us. Um, torjio interoperability. Um this this one is also quite important for us because we are now able to run multiple uh quantization algorithms like HQ and we also added an experimental um feature for doing um JavaScript and browser support uh via Wom and uh um yeah there are multiple other features that we've added but um without further ado I'm actually going to pass it to Min so that he can go through some of the uh demos. Thank you, Bill. [clears throat] Hey, my name is Morgan. Um, so as Bilan said, Executive is a general purpose uh runtime, you know, whatever you can write in PyTorch. PyTorch uh it can be delivered through export with minimal work. And these are some of the example models we have you know the LLMs, the computer vision models, the speech models and so on so forth. Uh so let's just maybe take an example of a multimodality. If you think of multimodel it's the architecture is relatively standardized now. There's the audio encoder or vision encoder. There's the projection layer and maybe a decoder layer. And uh we are basically trying to standardize it along with the hacking face transformers library and be able to export it to exeutor in a very compatible way. So you [clears throat] can use a oneline script to run it. For instance, Voxil is an audio modality uh audio input modality you can export to exech and on top of that you can just use that in in your runtime directly using the you know the three types of API Java C++ and um and Swift and iOS um uh we have a few demos um let's just pick this one uh so this is running Gemma 3 4 billion parameter this is vision and um uh vision and text input model and you know what it's doing is that it is taking a image as an input as as well as a text as an input and then it is running on X and pack on ARM CPU um with with CLI enabled uh on this one um so [clears throat] uh you know this is a memory bound uh memory bandwidth bound uh so as a result in the prefill is is is by fundamentally it is slow but uh you know when you all right okay right >> there we go >> so once once it actually is able to prefill and the tokens decode is uh is roughly eight to nine tokens per second. All right. Uh there will be a few these kind of demos uh throughout the throughout the Oops. So this is a Voxil. uh so describing audio I don't have the audio file this I'll I'll float it into the slides later today but you know you are trying to basically speak to the to the phone and then it's describing a a the [clears throat] describing the the interaction between two people here. So, this is a comedic conversation between two men, possibly friends. So, um talking about tattoos, you know. Yeah. All right. Oops. [snorts] Um, [clears throat] uh, Laura, this is a a common request we've heard from people. So, the idea is that you basically create a foundational model. Let's say you use llama as a foundation model. And then you have two data sets. You fine-tune it. Let's say in this example, one for Nobel Prize winners from 2025. Llama was not like this is pre after Llama launched. So, it doesn't llama doesn't have the information. and another data set for executive tutorials. And in this case, we are showing um you can load two Laura adapters together simultaneously without um incurring [clears throat] extra memory or binary size um load on the code. So, oh, so what? So, yeah. So here, so this is the the adapter specifically for that is trained for fine-tuned for executive tutorials. All right. Uh all right. So um these are some of the demos we have. Uh you know you can go to our website and look at the more of the success stories and demos. And I want to talk a little bit about you know what we are thinking for 1.1 and beyond in the future. And you know we worked so hard until now but you know we still want to actually uh do more [clears throat] things uh in the future. The first one is you know we have 12 backends as of today and eight four of them are production [clears throat] ready. So we would like to actually work on continue working on the eight uh remaining back ends to be production ready and uh on top of that you know ecosystem. So we executtors we think of it as a low-level library and I do think u natural step is in there where the middle layers or higher level program um higher level like middleware layers you know [clears throat] uh that there's more more uh for instance modeling libraries or SDKs can be integrated exeutors directly there. So maybe in the robotics space in the IoT space the mobile space right so uh uh the next one is um desktop laptop you know we've been talking about you know the edge edge edge you know one thing uh that's the big elephant in the room is like all right can you run on gaming PCs right can you run gaming PCs with CUDA CUDA back ends and a lot of people are doing this kind of local type of inference on gaming PCs um you know [clears throat] we've seen with like inspired by llama CPP and and MLX you know these are great runtimes that we've learned a lot from it uh so executive don't have uh kuda but until now uh we have and [clears throat] so let me just show uh maybe dancing in the masquerade I truth in plain sight ated pop roll click shots. Who will I be today or not? >> But such a talent as moving seamless was assumed. So this is a poem uh and uh [clears throat] we are basically running this uh poem in oh let me just uh running this poem in on cons in Windows on RTX 580 uh with completely no C++ completely no Python dependency or lip torch dependency and then the prompt you are asking is like how is the speaker feeling right Now it's specifically chosen because you know compared to whisper for instance you know uh you can't do that uh through whisper it doesn't convey emotions but with this demo we were able to uh [clears throat] retrieve a bit about the emotions of of the of the audio. All right so we have three minutes so let me uh so these also demos will be available in the portal. So um Java 3 is available as well. We also enabled uh uh whisper uh whisper directly here. So what is the just this morning we enabled it. So uh you know Mr. Quilter is the apostle of middle classes and we are glad to welcome his gospel. >> So this is a transcript. Uh so let's uh see whether it is able to transcribe on on a on a consumer laptop. Yeah, the Mr. Quilter is an aost apostle to the middle class. So yeah, it is able to uh transcribe pretty quickly. uh um yeah so we have numbers so far uh this is by no means a final number this is we've been you know if you for instance look at it it's 50 tokens if it's quantized 90 tokens close to 100 tokens for whisper 170 tokens but I do think uh this is a uh very early but very promising direction so the idea is that you know this is a model agnostic. Uh so instead of focusing on only LLMs or only um [clears throat] vision, you can actually do anything uh model agnostic like whisper is available, reset available. So we have demos on these and stay in the PyTorch ecosystem. Uh no P python runtime is required. So you can actually run in this native C++ um environments. No lip torch. So you can save on on binary size. Um yeah one thing that you know PyTorch already has the CUDA back end available. So we do not want to reinvent the wheel. So we are using the compiler technologies directly and actually interoperate with execut uh and you know validate on Windows and Windows WSL and Linux. So far uh yeah it's still early but very promising direction and we're so excited about this direction to be going into this new market of AIPCs. Um uh [clears throat] also we're currently experimenting on similar approach on MacBooks too. So leveraging metal GPU uh by using AOTI. So this is another example where Voxra [clears throat] example of running uh uh Voxrol but you know use the kernels from the PyTorch core and use it directly [clears throat] from the NPS metal back end and compile its uh um compile it and be able to do inference. So this is audio that captures identity chain. So anyway, we have another uh numbers for this still early. I just enabled it yesterday. Thank you, Manuel. Uh all right, so uh I just want to step back. You know, we talked about executtors 1.0. really this is uh we mobile and embedded is really the bread and butter of uh of we are going to continue working on mobile and uh mobile and embedded systems and I just want to say first of all thank you to all the partners you know I see so many familiar faces in this room and I glad that you know because of the collaboration we are able to you know successfully announce executive GA and also want to thank the teammates of PyTorch Edge for working really hard to come to this uh place. Thank you everyone. [applause] [applause]