PyTorch Mobile - David Reiss

Key Takeaways

break was canceled in exchange you get me hi everyone I'm David and I am here to talk about the PI torch mobile experimental release so what is PI torch mobile well it's pi torch for mobile I know that sounds a little bit silly but a lot of the questions we get when we talk about pi torch mobile are did you build a brand new framework are you going to have to or are we gonna have to export our model to a new format are we going to have to port a bunch of operators and the answer to all of those is no so no new framework no model conversion no operator porting this is just PI torch built for Android and iOS and we've made a lot of internal changes to make it work better on mobile but it's still the PI torch that you know and love and we believe that having a unified framework is the best way to enable you all to bring your work from research to production now one decision we did make is not to put a full Python runtime into our mobile build that would have been a little bit crazy so that means that eager mode execution is out of the question on mobile but what we do support is any torch script model so you learned about torch script earlier it's a large and growing subset of Python that can run most Pi torch models and that's the technology that PI torch mobile is built on so if your model is compatible with Lib torch using either tracing or scripting then it can run on mobile and that means that we support all of the control flow constructs and data types that are supported in pipe in torch script so oops yes functions yes tuples yes even name tuples also yes so that all sounds great let's talk about how to use it in case you weren't paying attention to Michael's talk this morning this is how you write a model with torch script and if we zoom in on this section down here it's really just these two lines of code that you have to write and that's it that's going to save your model to a file you can hand that off to a mobile engineer now mobile engineers you want to know how to run this so let's talk about that first you need to add a dependency on PI torch that's a one line configuration on both Android and iOS and that's gonna bring in a pre-built binary that's configured for development now you need to write your code to run the model so let's look at that on Android we have a Java wrapper for Lib torch and that means that you won't have to use the Android NDK you don't even need to install the NDK and I know the NDK is a lot easier to use than it used to be but it can still be intimidating for app developers who are used to programming in Java so we've taken care of all of that for you I just write Java code and this is what it looks like you have your feature vector you wrap it in a tensor you pass that to your model and you extract your data from the output so pretty simple alright now on iOS the API that we expose is the Lib torch C++ API and this gives you the maximum of flexibility and functionality but it means that you're going to have to write a small amount of objective C++ code to bridge between your app which is probably written in objective-c or Swift and Lib torch which is written in C++ but we think this isn't too significant of an overhead because it uses the same tool chain and the same build process as the rest of your iOS app so I'll go through the code if you've used Lib torch before it should look very familiar again feature vector wrap it in a tensor run it through your model and collect your output so it's basically the same four lines of code on both platforms to run your model and I want to show you what it looks like for real so we have some demo apps that we're going to be showing in the demo area this evening there's one on Android one on iOS the first model in them is a text model so you type in some text and it tries to predict which section of reddit that text came from the second model is a vision model that's trained on imagenet so it takes the live image feed from your phone camera and it tries to classify what it thinks it's looking at based on the 1000 class is an image net and in terms of model architecture the text model is based on a multi class classifier using an L TM that we've been testing in one of our production apps at Facebook the image model is based on a quantized mobile net B to Architecture and this really emphasizes the benefit of having a unified framework you can design your model train your model test your model with quantization on the server side and then you can use that same model on mobile because we have the exact same quantization framework same interface same semantics so though there are and there are other PI torch features that will be getting the same treatment so for example you heard about the deep type conversions this morning type promotion this morning and that works as well on mobile and there's also a name tensor which is going to be coming to torch script soon and when that's in torch script it will also be available on mobile so the code for both of the demo apps is available on github right now you can check it out you can use it as a starting point for your own integrations and just as a data point the Android app is a hundred percent Java the iOS app is about 85% Swift so just to recap what we're releasing today full torch script support I want to emphasize that it is the real torch script interpreter that's running on the mobile device there are pre-built binary releases already out in J Center and cocoa pods one line of configuration to install the Java bindings are bundled with the Android version and these releases contain all of the forward operators that have CPU implementations so it should be able to run any of your models because it's basically everything in the kitchen sink type of release now some of these models have some of these operators have already been optimized for mobile the one we paid the most attention to is convolution and the float version has some optimizations that we brought over from cafe to go which is our existing production deep learning framework for mobile and the quantized versions are based on Q&N pack which is a library of highly optimized quantized numerical kernels that we released about a year ago now not all the operators have gotten this optimization treatment so that brings us to what's coming up first and foremost we want to make things faster we think we can squeeze a little more water from the stone of these high-value operators but also find some of the operators that haven't got enough attention and make sure they're running up to speed as well we want to make the release smaller so that the current sort of kitchen sink build weighs in at about five megabytes per architecture and the biggest tool that we have for making that smaller is giving you all the ability to build a customized release of Pi torch mobile that is tailored for your own operators so for example if you're releasing an app that has a bunch of vision models you don't want to be bundling a bunch of code that has text operators in it that's just a waste we're experimenting with a wrapper for Objective C and Swift that we think can lower the barrier to entry for iOS developers even further and we've talked about doing a Kotlin wrapper but that I think is gonna be movin more by community demand because the Java wrapper actually does work pretty well either from Colin there are some more long-term projects as well when I add GPU support so if you've ever programmed a mobile GPU you probably know it's not quite the same it's just running some CUDA kernels but most mobile chips do have a powerful GPU and we want to be able to take advantage of this to accelerate your models and then even even further down that spectrum is neural network accelerators which are starting to show up on some of the newer mobile SOC sees and they can provide even a greater level of performance but often what comes with that is an even greater level of challenge when you're trying to program program them and so if we can sort of hide some of that complexity for you and make it easier for you to run your models on on those parts of a chip that would be fantastic so that's a quick look at what we're planning in terms of what you can do today please visit the mobile page it's linked right from the top of pite org try out the code if you're feeling brave you can ship a production app with it this is an experimental release but we do expect it to work but more importantly tell us how you're using it you know come to the PI torch discussion forum under the mobile topic post about your app let us know what's going well for you and even more importantly what you would like to see improved because we have a lot more work to do and your feedback will help guide our designs and also our priorities thank you very much [Applause]