TensorFlow for JavaScript (TensorFlow @ O’Reilly AI Conference, San Francisco '18)

Key Takeaways

how's it going everybody here to talk about tensorflow and JavaScript today my name is Nick and this is my colleague ping and we work on tensor flow GS here at Mountain View so the traditional thinking is machine learning only happens in Python right that's that's kind of what the you know everybody thinks about but is that always the case has anybody seen this before this is there's something we host on on our tensor flow documentation this is the machine learning playground the tensor flow playground and it was actually built by our colleagues in the East Coast and it was just a visual to put into some of our ml classes and it kind of shows how data flows throughout a connected neural network with different activation functions and this was a really popular project we built and it was a lot of fun to make and we've gained a lot of traction from it so we started to think maybe it makes sense to do ml in the browser there's a lot of opportunities for doing ml directly in the browser we don't need any drivers there's no CUDA installation or anything you could just run your code the browser has a lot of interactive features especially with over the last several years of development there's access to things like sensors and cameras easy you can easily hook up to that type of data stream and the other great part about doing ml directly in the browser is it's a good privacy use case you don't have to send any user facing data or any user data over the wire over an RPC to do inference behind the scene in your infrastructure you could actually just do that directly on the client so coming back to the tensorflow playground this is about 400 lines of JavaScript code it was very specifically typed for this project so our team kind of took this prototype and started to build a linear algebra library for the browser this project was initially started it was all open source under it was called deep learned is and we took deep learn j/s and aligned it with what we're doing tensorflow internally with eager execution and that type of alignment and launched tensorflow GS last April already and once we launched it we had a lot of really great community and Google built products and I want to highlight a couple this is one that we built at Google it's called a teachable machine this is all done in the browser there's like three labels you can give what you're training in the webcam there's like a green purple and red and it sort of highlights how a basic image recognition model can run directly in the browser so this is this stuff all exist online you can still find it and the community built a self-driving car all in the browser called medic car and this is cool you can watch a train and learn the inference and what the cars driving people built games so this is a this is a web game that somebody trained with tensorflow GS to avoid it's kind of a funny animation but there's a little dude running back and forth and he's hiding from those big balls and that the the model is learning to avoid the balls all through using tensorflow GS and continuing to play this one's really cool this is a Google project called magenta which does a lot of ML with audio we have a large library called magenta Jas which is built on tensorflow GS to do in browser or audio this is a cool demo somebody built it's a digital synthesizer that learns how the plate music and can drive with it another another cool example that just came out is this is all community built open-source it's called face api GS so it's a library that sits on top of tensorflow GS has a few different type of image recognition and can detect faces and facial features so even like toddlers work pretty well so I want to kind of showcase how our library pieces together there's sort of two main components to tensorflow Jas there's a core API and on a layer C API and that is all powered in the browser by WebGL that's how we did the linear algebra aspect for the browsers we bootstrap all the linear algebra all through WebGL textures and on the server side we actually ship our C code that we run Python or I'm so that power is tensorflow Python so it's you get the high-end CPU GPU and then eventually we're working on the TPU integration story for servicing and those who have used Kerris the layers api is almost the same as Kerris very similar syntax the core API is our op level and you'll anyone who's worked with tensorflow save models that api will be pretty somewhere ok what can you do today with tensorflow GS well you actually just author small models directly in the browser there's a limited amount of resources the browser's have so we kind of get into that a little bit later but right now you can do pure model training in the browser you can import pre train models so this is a model it's been trained somewhere else usually in the cloud or on some Python device and we have a tool to serialize the model and then run that inference in node or on the browser and we have the ability to retrain models so it's a very basic transfer learning we can bring in a model anyone who's seen tensorflow for poets it's a very similar exercise so the get started with the core API I want to do just a very simple basic fitting a polynomial so this is a scatter of some data we have and we're gonna write a really simple model to try to find the best fit for that this plot of data so the classic FX equals ax squared plus bx plus c excuse me so the first line this is all es6 style javascript for those who are familiar so we're gonna import at tensorflow /tf GS it's the name of our package and we namespace it as TF and our first step is to include three different variables a b and c and we actually initialize those as 0.1 this is gonna be passed into our training sequence the next option the next step to do is declare our function so this is all using the TF GS API for doing that f of X equals ax squared plus B to the power of X plus C and we have some sugar to make that a little bit more readable using chainable api so it's very common pattern in JavaScript next step is to declare a loss function just have a mean squared loss and then we declare the SGD optimizer with a default learning rate we've declared somewhere in this code and then finally we loop it through our training sorry box we pass through and every step we minimize our loss to the SGD optimizer this is a very similar to Iger style Python for those who have done that in the Python name next thing I want to highlight is the next step up that layers that carrot style API and to do so we've we've been working on doing audio recognition directly in the browser and I want to highlight just simply how that kind of works so really simple spoken commands like up-down left-right can be run through FFT to build a spectrogram so we take audio in and we build a spectrogram as an image and we train our model on that and we can actually build that convolutional net work pretty simply with our layers API and the first step is just the same as our fitting polynomial will include the package TF GS and then we're gonna build a sequential model this is very Charis Charis Charis style excuse me our first step is to do a comp to D a couple different filters and kernel size raivo activation functions again this is very Kerris very familiar for those who have used carrots then we have a pooling layer and then we're gonna go ahead and do some more comp 2ds in another max pooling level and so on we repeat as we work our way down the funnel and finally we flatten out our layers add some drop out add a large dense layer at the very end one more drop out layer and then finally our soft max for audio label audio labeling and finally let's compile the model so this is again very similar to Kharis we're gonna compile our model that we built we'll know any errors that we have as them as the model is constructed to give it an optimizer and we call model dot fit to start passing in our training data with our labels and once the model has trained we can save it to disk we have options for saving directly in the browser and on uhnot gs2 file and finally we can use that model to do prediction so we model dot predict and we pass in our spectrogram okay so those are two quick passes at some of the api's we use the higher-level core and in the lower level I'm sorry the higher-level layers API on the lower level core API but one of the cool parts of doing the browser is we can take in models that were even trained today and build interactive demos and for that I want to showcase a small video this was actually built by a collaboration of the tensorflow GS team and a Google internal design firm and we built this game for mobile devices and it uses mobile net to do emoji scavenger hunt so on the game the game will suggest an emoji and you have to run around the office and find it with your webcam on your phone and this is all doing inference it's PowerDirector browser for this one I'll play a quick video and it will kind of give you a better highlight of what's going on this is a slice of pizza so is this emojis have become a language of their own we use them everyday to communicate in our text and emails so much so that it's easy to forget about the real world objects they're based on which God is thinking can we create a game that challenges people to find the real world versions of the emojis we use every day introducing emoji scavenger hunt emoji scavenger hunt uses tensorflow j/s open-source meats machine learning meats javascript meats funds it works like this we show you an emoji use your phone's camera to find it before the clock runs out find it in time and you advance the next emoji while you're searching you'll hear a machine learning system doing its thing see if you can find all the emojis before the timer runs out why spy approve emoji scavenger hunt powered by machine learning start your search at gqo slash emoji scavenger hunt well I see a URL [Music] cool so they sort of show cases for someone's already done the hard work of training a model now we can build that great interactive demo okay so I want to highlight how we actually do that behind the scenes so the first step is taking that free training model this is mobile net it's been trained under Python tensorflow and kernel II in mobile net those who have used mobile net will know there's a object detector that you can tune for a specific labeling and then we import that into our JavaScript app - scavenger hunt first step once the models been trained we save it there's a few different password in this there's the traditional tensorflow saved model API and we also have support carrots as well so this is a sequential mobile net model for Kuras then we have a conversion step so this is a tool that tensorflow GIS ships over Python it's a pip install tensorflow j/s and you can use the tensorflow J's underscore converter for interacting with a save model there's a couple different options for finding the output of the inference graph and where we want to serialize our artifacts and then we also support the Charis style converter as well for hdf5 file format and finally we would like to load those artifacts into the browser so this is all JavaScript code for that save model it's TF load save model and we have two different artifacts that our script creates there's a weights link and then a link to the JSON file which describes the inference graph and again there's a Karass style one Karis actually ships all in one JSON file which has one down side of it avoids some of the caching that we do we provide for save models what happens in that model conversion steps so the first thing we do especially like save model has a lot of different paths for the graph there's a inference graph which the one we want there's steps for training and a lot of time if you're using the TF data pipeline there's actually graphs for all the data ops so we actually pull out the graph for inference and in class ops that's needed and run some optimization and the one other great thing we do for a save model is sharding of weights in the four megabyte chunks which cached nicely with modern browsers so it's only a one-time fetch for those larger models and we support about a hundred and twenty plus of the today's tensorflow ops and that convergence step and we're always adding more and again the TF cares layers are supported with this compression step I also wanted to showcase one more demo this is a newer demo that we've we've just shipped this summer and it's using Poe's net which is a human estimation demo and for this I'm gonna hand it over the pink who's gonna highlight this all right guys PostNet is not an example of converting Python training model and loading to the browser so on the right side you can see a lot of control that can fine-tune the model and on the left side is a live feed of a video so in the video you can see you can detect my face features as well as my you know body parts and so this is a collaboration between Google research team as well as external contributor so this model is in our model repository you can check them out on the left side you can see actually has about 15 fpm so frame per second you can build some cool application like build recognizing motions for sports etc we also have other models in that repository so like audio command model that Nick mentioned earlier and also we're adding some other like object detection model so all of that is available for you to use in the browser um you know just go ahead check him out and let us know if you'd do any quips Thanks the great part about that is it's feeding directly off of the camera feed in real time and we're doing about 15 frames a second and presenting over the USBC so does pretty well okay so I did mention earlier about training directly in a browser this is the retraining the transfer learning stub and for this we have another cool demo that we want to showcase again we're using mobile net and then this thing's gonna pull this demo up while I'm talking so we built this demo where we do we have a baseline mobile net model that we've loaded in the browser and we're going to Train ping space to play pac-man so he's gonna start collecting samples from the webcam of what his up down left is and so for this he's gonna use his face so as he's moving his face around he's collecting different samples that were gonna pass into that retraining step so there's a up down left and right and then with this demo as you hold down we were collecting more and more frames he's getting close okay and then now that he's collected is frames he's gonna click the terrain model button and we'll watch our last shoot straight down he only takes a couple seconds and now now he's ready to play pac-man so go ahead and hit play there pink all right this is this is a there you go so as his the model is running directly in the browser we retrained it to those pictures of his face and the controls are lighting up left down right based on what the model is doing so all right so this is a great use case of what you could do with taking advantage of some of the stuff that browser provides and doing accessibility for machine learning and building cool piece okay ping we can play pac-man all day man this these demos are all available on our site which will hike showcase at the end you can actually just run this today no drivers no anything to install okay cool all right so I've showed off a bunch of demos of doing using our core API using that layers API bringing in pre-trained models and doing some basic retraining so where does performance kind of step up stand for tensorflow GS for the our browser runtime that WebGL powered runtime so this is some benchmarks we've done using Python and mobile net so there's two computers that we use for these benchmarks the top one is a high-end workstation with a sheet 1080 GTX the high-end Nvidia card so it's super fast a little under three milliseconds for our inference time and then we used a MacBook Pro 13 inch MacBook with a non integrated graphics card or with a graphic integrated graphics card not a standalone graphics card and that was using the CPU built so there's no it's just the default AVX instruction step we ship with tensorflow and we were doing a little under 60 milliseconds for inference time so where does the tensorflow J estimate benchmark stand what kind of depends on that super beefy 1080 card we're really close about 11 milliseconds per inference time the CPUs are running on this laptop is a little bit slower it's a little under 100 milliseconds for inference time but that was still giving us that 15 to 20 frames per second which allows you to still build interactive demos so this discussion leads us to our next part which is we're just tensorflow GS and server side come in to we think there's a lot of great opportunities for going with JavaScript ml on the server side under no GS the ecosystem for no packages is really awesome there's tons of pre-built libraries off the shelf for NPM you can build applications really quickly and distribute them on all these different cloud services the default runtime for no GS v8 super fast it's had tons of resources put into it by come like Google and we've seen benchmarks where the GS interpreter in node is 10 times faster than the Python by enabling tensorflow with no GS we actually get access to that high-end hardware so those cloud TPU is the GPU and so on so those are all exciting things I wanted to showcase one real simple use case of nodejs and tensorflow the code snippet I have up here on the screen is actually a really simple Express app if anyone's used it it's just a request response Handler and we just handle the endpoint slash model which has a request and a response that will write out - so this model right now actually we have a model that we've defined and we're going to do some prediction on input that's been passed into this endpoint now to turn on tensorflow GS but note it's one line of code it's just importing the binding so this is a binding we ship over NPM and it gives you the high end power of tensorflow C library all executed under the note GS runtime and what can you do today under with server-side so all those stemless we showed of doing writing the model in the browser those actually just run under under node as well you can use our conversion script we ship the three major platforms Mac OS Linux in Windows CPU and then we also have GPU and CUDA for Linux and Windows we just launched Windows late last week and all the full library supports so the layers API in our core API all work to get today right out of the box with no GS and to kind of highlight how we can bring all these components of NPM and tensorflow GS and no GS together we built a little interactive demo so I know not everybody is super familiar with baseball but Major League Baseball advanced media has this huge data set where they record using sensors at all the stadiums the different types of pitches that players throat games so there's there's pitches that are really fast to have a high velocity and low movement and then there are pitches who are a little slower that have more movement so we we curated this data set and built a model all in tensorflow GS that trains against this data and detects I think seven or eight different types of pitches and it renders it through a socket so don't get too hung up on the intricacies of baseball this is just really solving a bread-and-butter ml problem of taking sensor data and drawing up a classification so for this I'll have ping run through the demo okay all right so for web developers you know you could really use tens of loaches to build a full stack kind of application so on the left side is a browser that I started a client on the on the on the browser inside a browser which trying to connect to the servers through socket IO on the right side I have my console are gonna start my server no Jess server immediately you see that is binded to our our tens of low CPU runtime and as it goes the models getting trained and the Train stats are feed back to the client side as training progress you can see the accuracy increase for all labels you know the curveball has about 90% accuracy right now with you know kind of service I implementation is easy to fini data not like inside a browser is much harder let me try to click this button what this would do is that will load live MLB pitch data into this application and we will try to run new foreign Sandow so let me click on that so immediately you can see the orange bar is the prediction accuracy for all of these labels some of them we actually get better with the live data it's 90 percent for changer something we did a little bit less accurate fastball to seem is only 68 percent so all of all I think is to demonstrate that the model actually generalize pretty well for the live data as well so yeah back to you I'm gonna actually kill that demo or my laptop will Dec great [Music] so just highlighting exactly what's going on there there was a the nodejs server which was doing our training there was a training data set and he validate data set which we were reporting back over socket IO how good we are at each class through our evaluation and then we add the ability to just easily reach out to MLB Vance media through node and parse through their data and then sending that to the model which was the orange prediction so kind of a cool use case of I've trained my model how does it stack up to real-world data and doing like a quick visualization and that was all playing JavaScript playing HTML and all the source code we've shown you today all the examples we've showed you they are open source and we'll link to them at the end here so performance so I highlighted where the WebGL runtime kind of stacked up with that Python benchmark so let's step in and look at Python benchmark against the node GS runtime so again these are those initial benchmarks I highlighted the node runtime itself is just as fast as the Python runtime for inference of mobile net this is because we're using the same library that Python uses and there's there's not much code to get to and then we're running on that high-end code so okay I've highlighted a lot of stuff we built since basically this year and launched in April the node stuff has been out since the end of May so what's next what's the direction that tensorflow GS is looking the going we have some high-level bets that we're doing there's a project that's going on that we're going to release you're very soon in the next month or so it's our visualization library so it's ability to pull in through the browser and do quick visualizations of your model and the data we have to look for that coming soon we also have a full data API so very similar to the TF data it'll be browser and node specific so there will be convenience functions for I just want to read data off of my webcam and not convert it to tensors how this API will provide that for you and on the server side it'll be giving high highly optimized data pipelines for doing nodejs terrain and so those are our two high level things those are the the big projects that kind of cross both of our runtimes looking forward for the browser we're working on performance so those benchmarks I should with WebGL a lot of them are the bottlenecks or limitations for a WebGL so we use 2d textures that render the texture data the tensor data there's some bottlenecks for downloading those textures reusing those textures so we're working on WebGL optimization we're also adding more and more ops lately that focus has been audio and text based models so we're adding a lot more ops to help with that we have a great stable library of image recognition ops and the audio stuff is coming and the other thing we're looking at is helping push that spec so the WebGL runtime was really interesting and it kind of helped bootstrap ML on the browser the WebGL isn't the best use case for this and we're looking at a few different options one is compute shaders which is much more similar to CUDA like where I can allocate the right amount of GPU memory I need to use and do that and we're also following closely the web GPU spec so there's a bunch of different offerings from Chrome and Internet Explorer and the browser vendors for what we want to do we're sort of helping watch that space and provide guidance as needed and on the node JSI cloud integrations a thing we're looking at this includes the serverless type integration points integration with our TP use and so on we're actually working on generating op code to provide all the core tensorflow ops and nodejs the Python version of tensorflow most of the code is actually generated from our off registry internally so we're we're writing that for typescript for JavaScript users too and we're providing a better async support with libuv so libuv is the underpinning and no GS for asynchronous programming we're working better to make that scheduler work much nicer so we're not blocking as much main application threads okay wrapping up showing you a lot of stuff I kind of want to step back and highlight a couple things first one is our core API that's the bread and butter of the tentacle OGS we it's the or op library allows you to interact with tensors and we also have our layers API which is our Kerris style api for training we also support saved model and caris model conversion today through our converter script and the newest runtime we have is no GS we've just got done talking about much of that and with that I want to thank you guys for attending everything I've shown you is on GS tensorflow org we have quite a bit of stuff up there there's all those demos that I showed you could actually they're linked in that page so you can find them as well as the source code we also have a variety of github repos everything we do is on github TF tensorflow /tf GS is our route one that's our Union package we keep track of all of our github issues there it also links out to a variety of things we have now we have an examples repository which has maybe ten to fifteen examples you can just run there's also a link to our model zoo so this is a models that we've pre train packaged up for JavaScript use and published over NPM a lot of them actually have wrapper API where you don't even have to take data and convert it to tensors and then pass it in for inference it says says here's an image HTML of canvas can you do a prediction so those are really cool all that stuff's linked on TFTs we also have a gallery to of community built stuff and it's it's always growing this is our community mailing list you could also find on our website there's a lot of good discussion of for how do I do XY and Z or I need this feature can you please help the gallery repo I just mentioned that's where all of our community built examples live and models repo and that's all [Applause]