Building with TensorFlow Lite for microcontrollers | Workshop

Key Takeaways

[Music] hi everybody and welcome to the tensorflow lite for microcontrollers uh workshop at this virtual i o uh i'm really excited uh to be here to talk about uh some of the fun stuff that uh the community and the team have been working on um and show you how you can do some really interesting stuff on uh tiny computers using machine learning i'd just like to introduce myself quickly um i'm pete warden uh i'm the uh tech lead for tensorflow lite for my quick controllers as part of the tensorflow team uh i've been working on tensorflow since it started and i've also been working on tiny ml and running machine learning on embedded computers for the last few years um and uh together with my co-author daniel uh i wrote the tiny mail book for o'reilly which talks about how to do a lot of the things that we're going to be covering today i'm also uh being involved with the tiny mouth foundation which helps put on a lot of really cool meetups um at the moment um but i bet you can find one in your area they're all over the world now and we also have an annual um conference that you can join uh that uh is a really good chance to meet uh lots of other people interested in all these kinds of work and as you might be able to tell i really love tiny computers i um just find something fascinating about having uh little devices like this arduino um that i'll be showing you and many even uh smaller devices that are tiny cheap um use very very little battery power so they can be powered off batteries um and i'm just really fascinated by all the things you can do with them so i'm really glad i'm getting a chance here to talk about some of them with you and the goal of this workshop is to give you a flavor of what we're doing with tensorflow lite from microcontrollers and the ways that you can use it to really get started with this whole field of tiny ml um you're going to uh if you make it through this workshop you're going to learn how to play on some invisible drums using the arduino learn how to control a web browser using your finger and how to actually train your own tiny uh accelerator driven motion model and deploy it to an arduino as well as that uh we're actually um going to be uh doing a con challenge for people um to try and see some of the cool things that the community can create with this technology and you'll get a chance to be featured on the google channels um and really we're hoping we get a chance to kind of show off some of the really neat things that people can do with this technology and while i'm talking about participation one of the things i really want to encourage you to do during this workshop is ask questions i really think that a lot of the best learnings that you can actually get um out of a live event like this uh is getting answers those questions you might feel embarrassed you might feel like oh this is this is kind of an obvious question but i bet that there are a whole bunch of other people out there who are uh wondering the same things so please um ask questions we have moderators online there's a button where you can actually ask questions um put in your questions the moderators will get them to me and hopefully we can actually you know cover some things i might not have covered uh because you're asking the right questions um so just to get started at a really high level um you've heard this be say tiny ml um what is it all about so from [Music] my perspective um i helped sort of put together this very um technical you know definition sort of little the app academic um which you can find on the tinyml.org website um but the key things you should take away from this are what i've um put in bold it's about using machine learning um it's about dealing with sensor data all of these microphones and accelerometers and uh tiny cameras that you get on these embedded systems and it's about running at extremely low power usually less than a milliwatt and it's really that low power requirement that is the most distinctive thing about what we're doing here because phones and other devices even whilst we normal raspberry pi's not the new picos but normal raspberry pi's they they can easily take um a lot of power or more to run um and we're talking about devices that can run uh in a thousandth of that power and so they're able to run on batteries of solar power for a long time um so why is this important why am i so excited about this stuff um having computers that are able to do interesting things that are really tiny and hardy means that these computers can end up in all sorts of places where you'd never put a phone or a laptop um or you know a server or some kind of uh cloud machine you can actually have these in environments that are extremely hot wet um out in the middle of nowhere where you would never normally be able to do similar stuff with more traditional computers and having these devices able to run machine learning means that you don't need to have um a multi-million dollar cloud data center or pay a lot of money for gpus in order to start using machine learning machine learning can be democratized if you can run it on these really cheap uh easy to get hold of devices and one of the key things that i mentioned before as well is that low battery consumption um if you think about all of the devices that um traditional computing covers whether it's mobile you know phones where you have to plug them in every night a person has to plug them in um or clouds cloud or laptop computers where again you have to be plugged into the mains either all the time or um frequently uh which means you have to have an environment that's been uh set up for that um if you have really low battery consumption or solar energy powered or some other energy harvesting then these devices can be completely independent of people you couldn't have a whole bunch of devices um you know if you imagine having to charge devices like you charge your phone um you can't have more than like one or two devices like your phone uh because you're otherwise you'll just end up spending all your time kind of plugging these devices in or you'll be running around changing batteries all the time if you can get this really low battery consumption then these devices become independent of us they don't need us to do constant maintenance and care them so that's one of the underappreciated uh you know ways that these embedded devices make entirely new use cases possible and i also think there's something really important about data that can be processed on device is processed on device you don't have to send your data anywhere i'll be talking later about things like voice interfaces that can run entirely locally so you don't have to send audio data anywhere which really helps for uh latency uh and also helps for uh privacy as well uh you know you can make sure that any of the data that's being used doesn't ever end up going anywhere and what i have been really fascinated by um and the reason i came into this area was when i first joined google back in 2014 um i spoke to the team behind the uh hot word the wake word recognition for um google's voice interfaces uh i won't say the phrase because it will wake up a bunch of my phones and probably a bunch of your phones but internally we refer to it as okg to avoid twigging all of that stuff but what really blew me away was that they had models that were only 13 kilobytes in size so they were using these really tiny minute models to run on the always-on dsps that are in a lot of android phones um to listen out for that wake word so the battery wouldn't run down um and that really got me thinking okay wow what can we actually do if we can do something useful with a you know so useful with a 13 kilobyte neural network model what other problems can we solve with these really tiny models too um and that's uh sort of one part of the tiny ml side the other part is there's been this amazing explosion of really interesting hardware that can run in that sort of milliwatt-ish range it turns out that it's totally possible to do a lot of compute in a very small power budget uh from a hardware side it's a lot harder to do communication it's a lot harder to radio it's hard to have very much in the way of um memory but if all you want is lots of compute uh lots of multiply ads then turns out that's a comparatively easy ask for a lot of uh hardware designs and um there's been this explosion of really interesting hardware platforms in this space so tiny ml is really the intersection of these two and i'm going to be talking about and using an arduino board here as a demonstration but there are a whole bunch of uh really interesting uh hardware platforms out there uh the raspberry pi pico uh we work a lot with spark fun we work a lot with adafruit uh there's been some fantastic work on esp32 devices with expressive we've worked with cadence on the dsp side we work a lot with arm to and they contribute code and many many others so uh tensorflow lite for microcontrollers i'll be talking about is really designed to run across a wide range of all these low power hardware devices and this is where tensorflow lite for micro controllers comes in we have really been focused on okay how can we get some good models um some good application use cases so that we can run on all of these pieces of hardware now maybe you didn't believe me but we actually have a great video that google's creative lab have put together um that we're going to play uh in a second that uh gives uh another perspective with some rather uh awesome graphics so if we can play that video [Music] in 2015 we introduced tensorflow an open source library for large-scale machine learning it was a big deal for us it was also just big so we made it smaller and smaller which was big but then we asked can we go bigger by going even smaller so we took small and made it tiny which was huge because now tensorflow fits on a computer this big that costs this much runs for this long without connecting to this it's tensorflow light for microcontrollers you can use it to track motion in sports maintain and monitor your crops or discover the depths of the ocean the possibilities are endless we can't wait to see what you do with it [Music] i absolutely love that video um it's uh by the google creative uh lab uh mostly based out of new york um who've also done a lot of the experiments i'm going to be showing you um so how does this whole tiny ml process work if you've got any experience with machine learning which you don't need to dive in um but if you do this loop that i'm going to show you here is going to look pretty familiar you have to gather the data you have to design and train a model um because we're deploying on these devices that have such tiny amounts of memory just tens of or hundreds of kilobytes we need to quantize the model down to eight bits uh rather than floating point so that it will actually fit and run efficiently um and then we need to get it from the computer that you've been training it on and put it onto a microcontroller and i'm quite excited by what we've actually put together um with these experiments because i think it helps you do a lot of these stages um in a way that's uh a lot easier to get into a lot faster to start getting results than some of the traditional tutorials you might have found so that's a good lead-in to how can you actually get started um so don't worry if you don't catch all of these slides um if i'm gonna hang out on this slide for a bit um so that you can actually try and grab some of these urls but if you look on the uh the experiments with google site you can also find links to these as well i'm going to be working through the readme or the readme for installing this stuff i'm going to talk through in a few slides first and then i'm actually going to go to screen sharing and try and do all of the installation um and all the way through to actually running the demos and training our own um models uh live um so as you saw earlier anything can happen uh i hope you can uh wish me luck and keep your fingers crossed um but what we're going to be doing um is uh hopefully you have an arduino um the tiny machine learning kit is a great uh pack uh to actually get um though i'm also gonna be talking about a cheat code that lets you skip a lot of these steps by getting a board with a pre-installed a pre-flashed version so you don't actually have to go through some of these extra steps so i'll talk about that as we get through this but if you want to install everything from scratch uh get the tiny machine learning kit from arduino or just get any um arduino nano ble sense 33 board and we'll install the arduino ide um we'll make sure we can actually connect the board um we'll make sure we can actually find the board in our ports which can be a little tricky sometimes and uh we'll be installing the libraries that you need uh then we'll be grabbing the sketch and compiling and uploading the sketch and if you actually get the pre-flashed uh arduino nano ble device that i'll be talking about uh towards the end you can actually skip all of those steps all you have to do is plug in the uh the battery and go to one of these urls and the sketch that we're installing will actually start working immediately and i'll be explaining a bit more of the magic of how that works as we go through the installation process because there's quite a long time when you have to wait for the board installation to work so you'll have plenty of chance to see me um chatting about uh what's happening under the hood so now it's demo time the very first thing you're going to want to do is grab the arduino ide and depending on what platform you're on you can see you've got download options for windows linux and mac os and since i'm on mac os i'm going to click on this um i've already donated so i'm going to click on the just download i've now managed to start the arduino ide the next thing i need to do is install the board package if you have your arduino already plugged in you might see this shortcut this little message about installing uh the package um now if you don't see that don't worry you can just go to tools board boards manager and then you'll if you do a search for embed uh you'll see arduino embed os nano boards and this includes the arduino nano 33 ble sense uh which is the one we want oh where did it go there we go so make sure we got the latest package and then i'm going to click on install and this is going to take a few minutes so while this is happening i'll tell you a little bit about what we're actually doing here um when i've been running workshops i've found one of the biggest obstacles to getting people up and running is that usb uart drivers which is what we're going to be using here through the arduino ide they're great when they work but especially on a lot of windows machines but even on some of the uh you know mac machines and linux machines and other setups i've had up getting the drivers uh to work correctly can be really really tough um and what's worse is that debugging why the drivers aren't working is really really hard so if i have a classroom full of um you know 20 students uh 15 of them will probably be able to have success with this but they'll probably be uh five who are just uh left stuck not able to uh get uh their uh projects off the ground even just to get started um because they're running into these um connection issues between plugging the board into the usb port and actually getting your computer to it talk to the board so looking for a way to try and make that easier so that people's first impression of doing this kind of tiny machine learning isn't this super frustrating process that's really hard to debug i decided to take advantage of the bluetooth low energy connection that's available on this arduino nano board and a lot of other modern connected uh embedded development boards and the other key ingredient is that chrome and edge browsers actually offer a web interface to talk across bluetooth low energy to any devices that uh the user connects to so by bringing those two sides together um what we've been trying to build is um a device that all you have to do is plug it in and then it runs a sketch on the arduino board that sits there listening on bluetooth for a um website or any other thing that can talk ble but in this case all of these examples are using web ble and javascript through a website um to give it a model to run and also the board is able to transmit information about the gestures the accelerometer information about gestures that are being performed by the board so you've got this great two-way api um and the big missing piece uh that we're trying to uh sort out with the uh spark fun boards that we'll be talking about um you know will tell you how to get hold of uh towards the end of this talk is having that sketch that sits there and listens on bluetooth looking for some web uh uh site that speaks that language to actually um uh give it a model or receive some training data from it and so all of the experiments i'll be showing you are actually built using that technology so the good news is we've managed to get that board installed and if i go to um board here i can actually choose the arduino nano 33 ble and once i've chosen the board you'll see that the board shows up there um and hopefully you should now see that there's this slash dev slash cu dot something something something that's actually the one that you want and the very first thing i'm actually going to do is just make sure that i can upload a simple blink program before i do anything else and the arduino has this really nice set of examples built in so i'm going to choose blink this will load up this blink sketch i will try pressing upload so it will compile the sketch and hopefully send it over the usb uart connection that i was talking about having so many problems with and uh get it uh uploaded you can see it's just uploading and flashing and what we should see now is it's done and if you can see uh it's probably very small but there's a little led blinking on the board so that actually worked um if you are having problems uh there's actually uh a great faq for the connection problems uh here so um you can find it through the readme link to there's a lot of often like port um and other issues uh that this covers but um thankfully this worked first time for me so now what i'm going to do is install some of the libraries they actually need and the libraries are listed here arduino lsm 9 ds1 which is the accelerometer library arduino ble so it can talk through the bluetooth low energy connection and the tensorflow lite for microcontrollers library and again arduino has this great system where you can just use the ide to install install these things so if i go to sketch include library manage libraries and go for i'm going to look for lsm the lsm 9 the s1 library and make sure i've got an up-to-date version just install that this is going to be a lot quicker than the boards uh installation thankfully um so i'm gonna grab the arduino ble library make sure that's i install this and i'm gonna grab the tensorflow lite library and install this as well and once that's finished um i'm then going to uh grab the grab the sketch and let's see if i can uh just find the right link to the sketch so and you can also see this getting started uh guide here which we link to all right just a second uh you can download the latest release you'll find a uh link uh here so if we go to uh this uh tf4 micro motion kit arduino sketch.zip then i'm going to open that up and you'll see that there's a dot inno which is the sketch suffix i'll open that up and with any luck this is what i was talking about when i said that there was a sketch that sits and listens to bluetooth and when it hears that a website is trying to upload a model to it um it uh will accept that model and it will also um and you can ignore the uh warnings here they look scary but they're just um harmless and what this does is it sits there and actually listens out for websites trying to connect and it also if a website wants uh it can actually get information about what's happening uh with the device um it will receive um kind of gestures that you're actually doing and we'll use this a little bit later on to help do some training so this compilation will take um again a couple of minutes [Music] you can you know take a look at the code all of this work that i'm going to be showing you is open source um so you can find uh everything that you need on github um and uh yeah you can feel free to uh dig in uh file uh bugs or pull requests on uh this code um if you're interested in the workings of this you can see within the loop function the very high level work that it's doing if it's in the middle of a file transfer it will be updating the file transfer um if it's um sending data back to a website it will actually do um imu uh connections um and you'll see it's flashing now when it's finished flashing you should see uh the eon ps here flashing red green and blue so i bring it closer hopefully you can see that uh and what that means is it's actually uh waiting uh for you to uh provide a um model it's waiting for a website to connect and tell it hey here's a model for you to run the nice thing about the pre-flash kits is all of this stuff i've done with downloading libraries and boards and connecting over usb and dealing with ports that's already been done if you get a pre-flashed arduino um then all you have to do is plug it in and you should see this red green and blue light flashing and once you're ready you're ready to go to some of these experiments i'm going to be showing you as i mentioned these experiments are websites because you can use javascript from a website to talk to this board which i think is pretty awesome uh you do need a browser like chrome or um [Music] edge and while it does work i've got this working as well from an android phone using chrome uh they're not on ios because that's using safari but on the android phone it works but um the model download is a little slower so enough talking let me show you what this stuff can do um the first thing i'm going to do is connect um and this brings up a dialogue that searches for any devices that support the protocol that we've defined for talking to this arduino sketch and you can see this kit actually shows up so i'm going to pair with it and down here you'll see that it's sending a model over to the device so next um i'm going to follow the video and it says hold the board so that the led is facing you and the usb connector is pointed down towards your wrist so i think i've got that set up and let's see how well i can actually play the drums uh so i'm gonna try an up high and fingers crossed this works there i go i'm going to try to the side and i don't know if the audio is coming through well but you should hear that it's actually playing drums so to the side and then download well i'm not going to um win a scholarship to juilliard with this performance but uh the good news is um this machine learning model is working um and you see it's not it's not perfect but it's not too bad and all of this code is open source this is trained using uh the trainer that i'm going to show you um as the last experiment so you can actually try this yourself next i'm going to show you a finger ui and in order to show this off i'm going to uh disconnect uh so i'm going to close air snare and hopefully i'll just wait and you should see the red green blue light flashing again so that means it's ready uh to actually uh connect so for this one i actually need to use a little rubber band to connect this and luckily i have one here i believe with the kits that you'll be able to get your hands on uh you will actually uh be able to use a more ergonomic uh strap a velcro strap i think they've they've arranged but i'm gonna just cut off blood circulation to my finger for a minute or two um in the interests of science um and hopefully uh you can see that i have now slapped this to my finger uh i'm going to uh connect and you'll see that same dialogue again and it's sending the tensorflow model and then i'm gonna uh have to remember i'm just keeping an eye on the different uh making sure i can remember all of the different uh gestures here i think i think i have them um so hey there go left left right right then i'm going to try clock yeah ah there i got cluck but it came off my finger that was my fault i'm 12. yay so this is really fun sorry i should uh i should uh let you have a play with this uh rather than me having all the fun uh but as you can see uh there's quite a lot of uh different things that you can do with uh different gestures that you can actually recognize with this um and you may be wondering okay how did we create this well i'm going to close this window and then move over to the tiny motion trainer and again i'm just going to wait and make sure that the uh board has now uh gone back to flashing red green and blue which means it's ready to start um i'm gonna take this off my finger so i get some some blood back and get rid of the uh this rubber band um and i'm gonna choose uh to start a new project and this is using tensorflow.js and you can see here you've got instructions and we're going to be capturing some data and then we're going to be training a model using this data and first thing i'm going to do is uh pair the device these defaults should be good at least for this initial go i'm then going to add a couple of labels uh i'm gonna be uh actually i'll call this one sideways and then i'm gonna start recording some gestures going sideways you can see i'm just flicking left right and i'm going to see if i can capture uh 10 of these and if you get any wrong you can go in and delete them i'm gonna stop recording uh you can use the x button if you got if you had any that you didn't want to use um and sorry just checking my sounds like somebody's chatting me great um i'm gonna now do an up down gesture so i'm going to create this gesture and i'm going to go up down oh i have to choose it and then go start recording and then i'm gonna go actually this is gonna be down up oh and that first one i don't think was right so i'm gonna try it again ah so i'm going to stop recording there and now i'm hoping that i've got enough data to actually uh train a model so you'll see these uh graphs you don't have to worry about them too much if you just wait for the turning to finish um you can see it very quickly got to 100 training accuracy um that's because it has a very small amount of data so um if you were doing this uh for a uh you know a more a project that had to be more bulletproof um you would actually uh try it uh with a lot more um so let's see how this model does so it detected that i got sideways let's see again yeah and then up down so this is used this hasn't um uploaded the model uh to the board yet but this is just using the gestures that were captured and running it on the ml model infants on the tensorflow.js side here so finally we can actually download a quantized version of this and once it's finished processing you'll actually see something that you're able to use as an arduino sketch so if i open up this um you'll see that there's actually an example with a model.h file oh no i do not want you xcode i would prefer visual studio code or you can use arduino to open it too [Music] but this model.h is um a uh binary version uh c data array that contains the quantized model and you can actually use that in uh there's an included.ino which lets you flash and uh bum this um so i think that's uh all i wanted to do for the screen sharing that was the demo um what does this all mean the reason i'm working on this is because i'm really really excited about all of the different things we can actually build with this all of the different problem domains that this kind of work can actually be applied to everything from agriculture you can imagine having loads of tiny devices that um are in fields that are keeping an eye out for crop pests or crop diseases using cameras um and you know maybe even look at things like how well-watered the crops are and really help us grow food in a much more efficient way with fewer chemicals um one of the things that i remember from the before times uh before we went into lockdown was sitting in uh a meeting at work um and uh if it was a really boring meeting and nobody was moving in the meeting room the lights would go off because they're just motion triggered wouldn't it be awesome if you actually had um little cameras that were completely local but that could actually tell how many people were in a room and adjust the ac adjust the lighting and all sorts of stuff like that in the home and around things like health and fitness there were loads of applications there was a really neat um solely uh project called uh gamer that um went into uh kids uh soccer boots and actually uh could tell how much real world activity they were getting and then would reward them in fifa the game for that kind of activity i would love to talk to more toy companies i think there are some really neat things we could do here as you've seen the google creative labs have done some really good stuff around art and music i would love to get these devices especially the pre-flashed versions into high schools because i think it's a really great way of giving people a very different approach to machine learning and an easy way to get started and i love getting this stuff into things like oceanography and really helping uh wildlife and the environment um and i have to say as well i i love that business monkey in the uh little workspace uh icon there that was uh that was one of my favorite parts of the uh slides that the creative labs teams put together so to wrap up here i wanted to give you some pointers to some other resources you might find useful tensorflow.org is the central hub for everything around tensorflow all of the experiments that i've shown you together with links to the github open source code you can find on experiments.with google.com um there's a free course on tinyml that i helped put together together with harvard um and the edx team um and some of the fantastic educators there you can take it for free you you pay if you want to get a certificate but everything is accessible just doing it for free and i highly recommend it it's uh i think it's came out really well um and uh the arduino board that i've been mentioning here you can actually find it at this link so i would you know check that out and as i mentioned at the start there's this tensorflow microcontroller challenge we want to see what you can build with this technology we're hoping to see some really innovative cool fun useful websites and we're going to be putting up prizes of uh 2 500 uh five prize uh for the five winners that are selected as well as we'd love to feature your work um and if you go to this link you can actually find out how to uh get um some uh some we we're giving away hundreds of free devices um that you should be able to uh you know get pre-flashed and skip a lot of the steps that we discussed here ah the question is will the library work with other boards apart from the arduino especially esp32 um and other ides like platform io yes definitely we try very hard to work uh with as many boards as possible the nice thing about the machine learning stuff is that it's just math um so uh we don't actually have to rely on many platform specific things we do have some platform optimizations for things like arm devices or other platforms where we can implement optimizations or the hardware providers can implement optimizations but they all of this actually runs um with uh on you can actually find uh downloads for esp32 you can find downloads for raspberry pi pico um you can find uh you know there's a lot of uh we've done a lot of work with adafruit and sparkfun and a lot of these other uh devices that are out there and we're always keen to try and help uh people port uh to uh new devices so i think we're just coming up to the end of our time here um i'm just uh checking the door here there's a question about alerting me when it recognizes one of 154 classes of birds on a pre-trained model um 154 classes aren't necessarily too much but i'm guessing that the biggest problem there would be trying to gather enough data and try to come up with a model that works well enough in the field but my email is pete warden at google.com or i'm pete warden on twitter if you'd like to follow me i would be happy to you know chat more about any of these applications and there's a question here about training and retraining at the moment you can't uh train models uh in the field but you can run pre-trained models okay um so thanks so much everyone thanks so much to creative labs uh they've been fantastic um and thanks to the crew who helped put this workshop together you