TensorFlow Lite for Android (Coding TensorFlow)

Key Takeaways

[Applause] hi and welcome to coding tensorflow where you'll learn all about coding machine learning and AI apps using tensorflow I'm Laurence Moroney a developer advocate for tensorflow and today I'll be looking at tensorflow Lite and getting it to run models on Android tensorflow light is tensor flows lightweight solution for mobile and embedded devices and lets you run machine learn models on mobile devices with low latency quickly so you can take advantage of them to do classification regression or anything else you might want to do without necessarily incurring a round trip to a server it's presently supported on Android and iOS via a C++ API as well as a Java wrapper for Android devices on devices that support it the interpreter can also use the Android neural networks API for hardware acceleration otherwise it will default to using the CPU here you can see me running the app on my Android device and using it to classify a number of objects it's quite fun to play with and note how the interpretation of a mouse changes as I move the camera around or the first mug has a high handle which kind of makes it look like a picture so that shows up as a possibility finally my Google mug is broad and looks a bit like a mixing bowl and the app detects that so how does all of this work well let's first talk about the model the app is built using a mobile net model mobile nets are small low latency low-power models that are designed to meet a number of use cases such as object detection face attributes fine grained classification and landmark recognition what's nice about them is that there are a number of different ones that are pre-trained including the model at this link which works for image classification that is compatible with tensorflow lite download that file and you learn zip it to see two files a dot TF Lite file describing the model and a labels file describing the labels that the model is trained for as you can see because your model is already in the TF Lite format it's ready to be run on tensorflow lite so let's take a look at the api's for that in fact i would recommend that you stick with built models for the time being as tensorflow light is a Developer Preview it doesn't support all the operations of tensorflow yet and you might encounter some issues with unsupported ones when you're converting your model to the TF light format okay let's get down to looking at it on a mobile now first of all to use tensorflow light in your android app you need to include the tensorflow light libraries you do this by editing your build up Gradle file to include them once you've done that in synced you'll be able to import a tensorflow interpreter an interpreter loads a model and allows you to run it by providing it with a set of inputs tensorflow light will then execute the model and write the outputs it's really as simple as that of course you have to load the model and the API makes that very straightforward now a good place to store your model is in your app assets the code will then read the model directly from there it doesn't have to be there of course you can load a model from just about anywhere but make sure you get your powers right once you've done that you can instantiate an interpreter and in this case I called it tensorflow light and loaded the model into it let's look at the app again now what's going on here is that it's reading frames from the camera and turning those into images it then uses those images as an input to the model which in turn outputs values these values are an index to the appropriate label and the value for that label the probability that the image matches that label will pick the top three and will write them to the user interface let's take a look at the code first you can see is the input we're getting the image data which is received from the camera as a bitmap we convert that bitmap into a byte buffer which the model can read this buffer is called IMG data remember that we'll come back to it in a moment when you downloaded the model you also had a TF Lite file and a txt file containing labels these labels are a list of all of the objects that the model can recognize here's the first view so for example labeled zero is background label two is goldfish and so on remember the image data from a moment ago well we'll load into our tensorflow light interpreter and we'll get an output called label proper ray it's probably easiest if we look at that in the debugger so here you can see I had a breakpoint on looking at a picture of my mug as you can see the top elements unless the goldfish etc have very very low probability indeed it's like 1.7 nine times 10 to the minus 5 but when you scroll through the output lists you'll see for example that label 505 has a relatively high priority and number 505 corresponds to coffee cup which as you'll remember from the video that's what I was looking at if you want to try an app for yourself that does all of this you can get it on github at this link and that's a starter intensive flow light for Android this exciting technology will let you load all your models onto an Android device taking advantage of onboard hardware and allow you to execute them I showed an example using image recognition in a video stream but of course you're not limited to that at the moment tensorflow light is in Developer Preview so you may encounter some restrictions in operations that are supported but we're updating it all the time finally if you want to learn more including how to retrain the model I showed in this video to tailor it for specific scenarios check out the tensor flow for poet code labs on the Google developer site I can't wait to see what you build on mobile with tensorflow Lite and don't forget to hit that subscribe button for more great coding with tensorflow videos Cheers don't forget to click the subscribe button for more great videos like these what's