TensorFlow Lite for mobile developers (Google I/O '18)

Key Takeaways

thank you so much for coming to our session this morning I'm Sarah Sarah Javon I'm on the tensorflow Lite team and we work on bringing machine learning to mobile and small devices and later on I will introduce my colleague Andrew celli who will be doing the second half of this talk so the last couple of days have been really fun for me I've gotten to meet and speak with many of you and it's been really nice to see the excitement around tensorflow light and today I'm happy to be here and talk to you about all the work that our team is doing to make machine learning on small devices possible and easy so in today's talk we'll cover three areas first we'll talk about why machine learning directly on device is important and how it's different than what you may do on the server second we'll walk you through what we have built with tons of low light and lastly we'll show you how you can use sense of low light in your own apps so first let's talk about devices for a bit what do we mean when we say a device well usually a mobile device basically our phones so our phones are with us all the time we interact with them so many times during the day and modern phones come with a large number of sensors on them which give us really rich data about the physical world around us another category of devices is what we call edge devices and this industry has seen a huge explosion in the last few years so some examples are smart speakers smart watches smart cameras and as this market has grown we see that technology which only used to be available on more expensive devices is now available on far cheaper ones so now we are seeing that there is this massive growth in devices they're becoming in freezingly capable both mobile and edge and this is opening up many opportunities for novel applications of for machine learning so I expect that many of you are already familiar with the basic idea of machine learning but for those that aren't I'm going to really quickly cover the core concept so let's start with an example of something that we may want to do let's say classification of images so how do we do this so in the past what we would have done was to write a lot of rules that were hard coded very specific about some specific characteristics that we expected to see in parts of the image this was time consuming hard to do and frankly didn't work all that well and this is where machine learning comes in we with machine learning we learned based on examples so a simple way to think about machine learning is that we use algorithms to learn from data and then we make predictions about similar data that has not been seen before so it's a two-step process first the model learns and then we use it to make predictions the process of model learning is what we typically call training and when the model is making predictions about data is what we call inference this is a high-level view of what's happening during training the model is passed in label data that is input data along with The Associated prediction and since in this case we know what the right answer is we are able to calculate the error that is how many times is the model getting it wrong and by how much we use these errors to improve the model and this process is repeated many many times until we reach the point that we think that the model is good enough or that this is the best that we can do this involves a lot of steps and coordination and that is why we need a framework to make this easier and this is where tensorflow comes in it's Google's framework for machine learning it makes it easy to train and build neural networks and it is cross-platform it works on CPUs GPUs TP use as well as mobile and embedded platforms and the mobile and embedded piece of tensorflow which we call tensorflow lite is what we are going to be focusing on in our talk today so now we want to talk about why would you consider doing machine learning directly on device and there are several reasons that you may consider but probably the most important one is latency if the processing is happening on the device then you're not sending data back and forth to the server so if your use case involves real time processing of data such as audio or video then it's quite likely that you would consider doing this other reasons are that your processing can happen even when your device is not connected on to the Internet the data stays on device this is really useful if you're working with sensitive user data which you don't want to put on servers it's more power efficient because your device is not spending power transmitting data back and forth and lastly we are in a position to take advantage of all the sensor data that's already available and accessible on the device so this is all great but there's a catch like there always is and the catch is that doing on device ml is hard many of these devices have some pretty tight constraints they have small batteries tight memory and very little computation power tensorflow was built for processing on the server and it wasn't a great fit for these use cases and that is the reason that we built tensorflow light it's a lightweight machine learning library for mobile and embedded platforms so this is a high-level overview of the system it consists of a kind where we convert models from tensorflow format to tons of low light format and for efficiency reasons we use a format which is different then it consists of an interpreter which runs on device there are a library of ups and kernels and then we have api's which allow us to take advantage of hardware acceleration whenever it is available tensorflow Lite is cross-platform so it works on Android iOS Linux and a high level developer workflow here would be to take a train tensorflow model converted to tensorflow Lite format and then update your apps to use the tensorflow Lite interpreter using the appropriate API an iOS developers also have the option of using core ml instead and what they would do here is to take their train tensorflow model and convert it to core ml using the tensorflow - core mm converter and then use the converted model with the core ml runtime so the two common questions that we get when we talk to developers about tensorflow Lite is is it small and is it fast so let's talk about the first question one of our fundamental design goals with tensorflow Lite was to keep the memory and binary size small and I'm happy to say that the size of our core interpreter is only 75 kilobytes and when you include all the support adopts this size is 400 kilobytes so how did we do this so first of all we've been really careful about which dependencies we include secondly tensorflow light uses flood buffers which are far more memory efficient than protocol buffers are one other feature that I want to call out here intensive low light is what we call selective registration and that allows developers to only use the ops that their model needs and thus they can keep the footprint small now moving on to the second question which is of speed so we made several design choices throughout the system to enable fast start of low latency and high throughput so let's start with the model file format tensorflow light uses flatbuffers like I said and flatbuffers is a cross-platform efficient serialization library it was originally created at Google for game development and is now being used for other performance sensitive applications the advantage of using flatbuffers is that we can directly access the data without doing parsing or unperson of the large files which contain weights another thing that we do at the time of conversion is that we pre fuse the activations and biases and this leads to faster execution later at runtime the tensorflow Lite interpreter uses a static memory and static execution plan this leads to faster load times many of the kernels that tensorflow light comes with have been specially optimized to run fast only on unarmed CPUs now let's talk about hardware acceleration as machine learning has grown in prominence it has felt quite a bit of innovation at the silicon layer as well and many hardware companies are investing in building custom chips which can accelerate neural network processing GPUs and DSPs which have been around for some time are also now being increasingly used to do machine learning tasks tensorflow Lite was designed to take advantage of hardware acceleration whether it is through GPUs DSPs or custom AI chips on Android the recently released Android neural network API is an abstraction layer which makes it easy for tensorflow Lite to take advantage of the underlying acceleration and the way this works is that hardware vendors write specialized drivers or custom acceleration code for their hardware platforms and integrate with the Android and API tensorflow light in turn integrates with the Android and an API via its internal delegation API a point to note here is that developers only need to integrate their apps with tensorflow light tensorflow light will take care of abstracting away the details of hardware acceleration from them in addition to the Android and an API we are also working on building direct GPU acceleration in tensorflow light GPUs are widely available and used and like I said before they are now being increasingly used for doing machine learning tasks similar to era an API developers only integrate with tensorflow light if they want to take advantage of the GPU acceleration so the last bit on performance that I want to talk about is quantization and this is a good example of an optimization which cuts across several components in our system so first of all what is quantization a simple way to think about it is that it refers to techniques to store numbers and to perform calculations or numbers in formats that are more compact than 32-bit floating-point representations and why is this important well for two reasons first model size is a concern for small devices so the smaller the model the better it is secondly there are many processors which have specialized sim D instruction sets which process fixed point numbers much faster than they process floating-point numbers so the next question here is how much accuracy do we lose if we are using eight bits or 16 bits instead of the 32 bits which are used for representing floating-point numbers well the answer obviously depends on which model that we are using but in general the learning process is robust to noise and quantization can be thought of as a form of noise so what we find is that the accuracy is 10 to be usually within acceptable thresholds a simple way of doing quantization is to shrink the weights and biases after training and we are shortly going to be releasing a tool which developers can use to shrink the size of their models in addition to that we have been actively working on doing quantization at training time and this is an active area of ongoing research and what we find here is that we are able to get accuracies which are comparable to the floating-point models for architectures like mobile light as well as inception and we recently release a tool which allows developers to use this and we are working on adding support for more models in this okay so I talked about a bunch of performance optimizations now let's talk about what does it translate to in terms of numbers so we benchmark to models mobile lat and Inception v3 or the pixel tool and as you can see here we are getting speed ups of more than three times when we compare quantized models running on tensorflow light versus floating-point models running on tensorflow I'll point out here that these numbers do not include any hardware acceleration we have done some initial benchmarking with hardware acceleration and we see additional speed ups of three to four times with that which is really promising and exciting so stay tuned in the next few months to hear more on that now that I've talked about the design of tensor flow and performance I want to show you what tensor flow light can do in practice let's please roll the video so this is a simple demo application which is running the mobile net classification model which we trained on common office objects and as you can see it's doing a good job detecting them even this tensorflow logo that we trained this model on like I said it's cross-platform so it's running on iOS as well as Android and we also are running it here on Android things this was a simple demo we have more exciting demos for you later on in the talk now let's talk about production use cases I'm happy to say that we've been working with partner teams inside Google to bring tons of low-light to Google Apps so portrait mode on Android camera hey Google and Google assistant and smart reply are some features which are going to be powered by tensorflow light in the next few months additionally tensorflow light is the machine learning engine which is powering the custom model functionality in the newly announced ml kit and for those of you that may have missed the announcement mo kit is a machine learning SDK it exposes both on device and cloud powered api's for machine learning as well as the ability to bring your own custom models and use them these are some examples of apps that are already using tensorflow lite VRML kit picsArt it's a really popular photo editing in collage making app and visco it's a really cool photography app so back to tensorflow light and what is currently supported so we have support for 50 commonly used operations which developers can use in their own models I will point out here that if you need an app which is not currently supported you do have the option of using what we call a custom up and using that and later on in this talk Andrew will show you how you can do that up support is currently limited to inference we will be working on adding training support in the future we support several common popular open source models as well as the quantized counterparts for some of them and with this I'm going to invite my colleague Andrew to talk to you about how you can use tensor flow light in your own apps thanks Sara [Applause] [Music] so now that you know what tentacle light is and what it can do and where it can be run I'm sure you know you want to know how to use it so we can break that up into four important steps the first one it and probably the most important is get a model you need to decide what you want to do it could be image classification it could be object detection or it could be even speech recognition whatever that model is you need to train it yourself you can do that with tentacle just like you trained any other tensor flow model or you can download a pre train model if you're not ready to make your own model yet or if an existing model satisfies your needs second you need to convert your model from tensor flow to tensor fold light and we'll show some examples of how to do that in a second third if there's any custom ops that you need to write this could be because you want to spot optimize something with some special instructions you know about or it could be that you're using a piece of functionality that we do not yet support like a specialized piece of signal processing whatever it might be you can write your ops this may not be necessary of course the last step is to write an app and you're going to use whatever client API is appropriate for the target platform so let's dive into some code converting your tenth of a model once you're done with a tensorflow training you typically have a saved model or you might have a graph desk what you need to do first is put this through the converter so here I'm showing how to do this within Python so if you download the normal tensorflow to Lane that's pre-compiled like the pip you're able to run the the converter and it just takes a save model directory in or a frozen graph def and it you specify a file name of what TF Lite file you want and that will output a flat buffer that's on disk that you can now ship to whatever device you want now how much you get it to the device you could put it into the package you could also say distribute it through a cloud service where you can update your model on the fly without updating your core application whatever you want to do is possible so next once you've converted well you might actually run into some issues doing a conversion because there's a couple of things that can go wrong so the first one is you need to make sure that you have a frozen graft F or a save model both of these are able to get rid of the parts of the graph that are used for training these are typically things like variable assignment variable initialization optimization passes these are not strictly necessary for doing inference that is prediction so you want to get rid of those out of the graph because we don't want to support those operations right now because we want to have the smallest version of the runtime that can be distributed to keep your binary size small the second thing that you need to do is make sure that you write any custom operators that you need and now I'll go into a little bit of an example of doing that well before that let me tell you one more thing which is we also have some visualizers let you understand the model that you've transformed and the transformation process so take a look at those they're linked off of the documentation so let's get into writing a custom op so what kind of optimized we need well here I have an example that's a little bit silly but it's to return pie so the important thing when you write an op is that you need to implement for C functions so we have a C API for defining operations and the reason we do this is that all of our operations are implemented in this way so they can run on devices that only support C eventually but you can write kernels in C++ in this case what I'm doing is I'm ignoring the input tensors and I'm putting an output tensor which is M pi now if you had input tensors and you wanted to make a an output tensor you could also read the input tensors and then say oh x 3 and now I have a x 3 operation this was going to be application dependent and of course as I said before you don't always need to do this I'm just laying this out here to show that if there's some functionality that you need we are extensible okay once you've converted your model you need to use a client API let me start with the C++ API but we have other language bindings as well that I'll get to and but in any of the bindings it's going to follow the same basic pattern the pattern is create an interpreter and load the model fill in your data execute the model and read back your data so it's very simple so in the C++ API the first thing you do is create a model object and this is given the file name of the tensorflow light file and it creates an object that is going to hold that model and M map it so as I said before we use flat buffers and the reason why is that we can M map the buffers which means that there is zero latency to start running the model effectively okay second if you have any custom operations you can register them so basically at this phase you're deciding which operations to include into your runtime by default we provide a built in op resolver that's that includes all of our default operations you might also use selective registration that we alluded to before where you include only a subset of the operations in this case you might provide a minimal resolver and if you wanted to use the custom operation that we had before you would create a custom resolver that would tell tensive row light how to find your custom operation so now we know what our ops are and where to get the code for them and we know our model now we need to create an interpreter object so we take the pair of model and resolver and put it together and it returns an interpreter this interpreter is going to be our handle for doing our execution so the next step is we're going to perform our execution but before we can do that we need to fill the buffer so if you have a model like a classification model that is something that takes an image in where are you going to get that image well the obvious place you might get it would be from your devices storage if it's an image file name but also commonly would be a camera whatever it might be you produce a like a buffer in in this class it's gonna be a floats hour and an in star buffer and you fill it in to our buffer and once you fill this buffer you're ready to run so we filled our buffer tensa for light has all the information it needs to run the the execution and we just call invoke now it's going to block until that execution is done and then we're going to be able to read the output of it in and in an analogous way to our input so that is we can get a float star buffer out which could represent the class numbers and then you're free to do with that data whatever you want so for example in an image classification app that we showed before you would read that index out map it back to a string and then put it into your gooeys display great so now we know how to use C++ what if you're using another platform for example raspberry pi on raspberry pi the most common thing to use is probably Python and again it's gonna follow the same basic pattern first we create an interpreter object the interpreter object is now our handle how do we feed data well since it's Python we can use numpy arrays and this is really convenient because if you'd need to do pre-processing or post processing you can do it with the primitives that you're familiar with and this is kind of a theme that goes on that we want to keep our bindings as idiomatic as possible in the language that they are and also keep it performant so in this case we put in some known PI ray and we take out some numpy array so that's Python what if you're writing an Android app or you want to write an Android things application then you might use the Java API so in this case it's the same thing you take you build an interpreter give it the file name of the interpreter it might be from a resource if you're doing an Android application and then finally you're gonna fill the inputs in and and call wrong so one of the things that we did for the Java API is that we know that many Java programmers don't really want to deal with building their own native library so in that case you can just use our Gradle file here which will include our pre compiled version of tensor faux light you don't have to download our source code and even for the tooling parts where you do the conversion from tensorflow to tensor of light you can download the pre compiled version of of tensorflow as i alluded to before great so what if you're doing iOS well in that case you can use the C post will say be API you can also use the Objective C API but again we a precompiled binary in the form of a cocoa pod okay so now that you know how to use tensor for life I want to tell you a little bit about what's going to be coming up intensive row light one thing that we've haven't been asked for a lot is adding more operations so the more operations that we add the more models can be run from tensorflow out of the box the other thing that happens with machine learning that's often difficult is that researchers come up with new techniques all the time and that means that tensorflow is always adding operations that means that we're going to continue to follow tensor flow as it adds important operations and add them into tensor for light as well okay the second thing we're going to do is we're going to improve the tooling provide better documentation and tutorials and try to focus on ease of use so it's really easy for you to understand on end-to-end examples how to integrate tensor flow life and the third thing which sara already mentioned but I'll mention again is that we're excited about on device training on device training is really exciting because it allows us to refine a model based on a user's experience it allows us to decouple that experience from going to the cloud so if they're disconnected we can and continue improving the model so there's a lot of requests for this this of course will require more and more computation on the device but we're excited about upcoming upcoming hardware accelerators that will make this more and more possible okay one more question before we get into some exciting demos when should I use 10 circle light so as we alluded to before we're starting to use tensorflow light for our first party applications and thirty third party applications are also using it that means that what we're doing moving forward is that we're going to make ten circle light our standard solution for running ml on small devices and mobile devices tensor fly to tensorflow light currently supports a subset of tensorflow ops this means that our recommendation is that you should use tensor fro light if you can and let us know and let us to know about any missing functionality you need it's not quite done you probably want to see our demos so with that I want to show you a video of retrained models we showed you that tensorflow go be logo being recognized this is a common theme that we get which is people like our pre trained examples like mobile net but they may not have an application where they need to tell the difference between five dog breeds and many zoo animals they might have an office scenario where they have markers and whiteboards and in fact as we were testing the app we found we had this issue - it's like we don't have the classes that are in these pre trained models so one of the great things that one of our other tensorflow members created was something called tensor full for poets and there was a codelab about that and it's available online as well and it basically allows you to take a pre train image model that has really good detector ability and put your own classes into it and I want to show you a demo app that we created that runs on the PC and creates ten to full line models for you so can we go to the video okay so we showed you before can we recognize scissors and post-it notes well let's try it out you always want to try these models check ok the scissors looks good right ok great post-it notes also looks good but what if we had another object an object that's you know more common more important like this metal tensorflow logo this happens in everyday life right ok let's go take a look at how this does well it's label this other that's not very good but the great thing about machine learning is that we can fix this and the way we fix it is we add data so we have our application we've gone to our training tab now and now we're going to define a class called tensorflow and this is basically short for tentacle logo and now from our webcam we're gonna click capture and we're gonna capture a couple of different perspectives as many as we can and ideally you would take it on different backgrounds so it doesn't associate the background with it being a tensor for the logo then I click the train button and right now it's using tensorflow to Train the model and you can see it's converging to a good validation accuracy it's going to reload the model and we're testing it intense or full light running on the PC right now and see that it's recognizing tents are full correctly so it's that fast and easy but also we can take that model and we can move to Android and iOS and use the exact same model and update it so thanks now let's move to a live demo so I'm gonna go over here to the podium alright okay so classification what I just showed you is kind of this idea that you have an image in and you put an image out and you put classifications out but what if you have multiple objects in the scene or you have something in a corner of an object you also want to know where in the where in the scene that object is and that enters this model called single shot detection it's a type of model and it turns out that our friends in tentacle research released a package called object detection as part of the tentacle models and that basically allows you to use their pre trained model that recognizes many classes so what I've done is I want to load that onto a small device in this case we're talking you a lot of things about mobile devices I want to show you another small device this is a Raspberry Pi so the Raspberry Pi is a really cool example of a device because it's very cheap and easy to get so any high school student could have one of these you can have many of these and just use them for a dedicated project but the other great thing about them is not only are they relatively powerful but they're also able to interface with other hardware they have GPIO pins and this this can be capitalized in a number of different ways but one way is to run Linux and that's what we're doing here but you can also use Android things which you can see one of the cool sand of the sandbox has many examples doing that so you could also do this with Android things so in this case I have the system the system board right here and it's connected to a motor controller and this motor controller is just a microcontroller that interfaces to servo motors these servo motors can go up and down left and right and they can basically aim the camera so now what we're going to do is we're going to load our object detection model onto this device and we're gonna actually run it in order to recognize different objects so let's go to the demo feed please and we can see my app you can tell by the beautiful nature of this app that I'm not a great app developer but this is what I can do on a weekend so give me a little bit of slack okay so here if we hold up to Apple it's recognizing the Apple and it's telling us you know what probability and where the object is now that's all good and fine but when we couple it with the ability to move where I'm going to turn on the motors now and now I'm going to bring back the Apple and what I'm going to do is I'm going to move the Apple in in the screen and it's gonna try to keep it centered so as I move this Apple it's basically going to try to keep it centered so it's like a little virtual camera person and this works on other objects like this banana here hopefully oh there we go and it's going to keep that Center and if you put two objects in the in the screen it's going to try to keep them both in okay so so we've got a little bit of false detection but basically it's going to try to keep them both Center so this is really a fun application and I bet you can come up with many different applications that are really exciting to do with this type of application so can we go back to the slides again so I like I said this is basically what I can do in a weekend but I imagine great app developers and people with a lot of creativity about connecting devices and connecting software can do many interesting things so what I want to do now is I want to tell you in summary tents are full light you've seen a lot about it basically we feel it tents or flight makes on device ml small fast and easy and we think that you're gonna find it very useful in all of your applications and we're excited to see what you build come talk to us I'm going to be in office hours at 12:30 you can come talk to me in addition you can come to our sandbox if you haven't already and we have of course the examples that I showed you here we have the the tracking camera we also have the object classification on mobile device but another cool thing that we have is the donkey cars and this was done by a group outside of Google and we converted them over to tensor faux light and we are excited to say that their application works really well with tensorflow light as well so with that I hope you check these things out I want to tell you that if you want to get started you can go to our documentation page you can go to the tensorflow door page and there is a TF Lite page where you can find more information about it in addition our code is all open source you can get it on github you can download it modify it submit a pull request and of course file any issues that you have while using it in addition if you want to talk about tentacle light talk about your applications ask us about feature requests please send to our mailing list this community is really exciting we found that in open sourcing tensorflow we got a lot of excitement we got a lot of interest and we made it a much better piece of software to use forever in both people inside Google and outside Google and we hope that you'll engage tenser fold light in the same way that tensorflow has been engaged so with that I want to thank you for your attention for coming to i/o for listening to our talk about tensor pro-lite I also want to thank you thank our Google partners this product didn't come out of isolation from all of our experience building mobile apps with machine intelligence as we gain experience we found that there was a common need and that was the genesis of tensor fold light so all of our partners provided application provided feedback even provided code and help with with our models so thank you so much to you and to them and enjoy the rest of i/o [Applause] [Music]