Superpowers with TensorFlow.js (TF Fall 2020 Updates)

Key Takeaways

hello everyone i'm jason mays developer advocate for tensorflow.js here at google which basically means if you're using machine learning in javascript there's a good chance our parts will cross now today we'll talk about how we can achieve superpowers in the browser with tensorflow.js so let's get started now the first question you might ask yourself is why would you want to do machine learning in javascript in the first place and javascript has many unique setting points to consider here so let's look at some of those first javascript enables you to use machine learning anywhere that javascript can run and that includes the browser server side desktop mobile and even internet of things based devices and if we dive into each one of these stacks in more detail you can see many of the technologies we know and love on the browser stack we've got modern web browsers server side is driven by node.js react native for mobile native apps electron for desktop native apps and of course raspberry pi for iot now javascript is one of the only languages that can run across all of these devices without extra plugins being required giving you the ability to deploy and run anywhere with one code base and that's very powerful stuff now with tensorflow.js you can run or retrain via transfer learning or write your own models completely from a blank canvas just like you can do in python with the original tensorflow but in javascript and with this you can use it for anything you might dream up things like sound recognition gesture-based interaction sentiment analysis conversational ai and much much more now the three main ways we can use tensorflow.js based on your familiarity with machine learning javascript or both and the first way is to use our pre-trained models these are easy to use javascript classes that can be used for many common use cases there are many situations where you don't need to train a brand new model from scratch and instead you can leverage existing work that exists so let's take a look at some of those here you can see several popular pre-made models available with tensorflow.js today things like object detection or body segmentation the act of classifying each pixel in an image to determine if it belongs to a human body or not or what about pose estimation to understand where the joints in the human skeleton may actually be there's various natural language processing models such as sentence encoding or albert q a model and many many more let's see some of them in action first up is object detection this model is using a model known as coco ssd behind the scenes and is trained on 90 common objects as such it can recognize various objects in images and provide us with the location of each object with the bounding box information it also exposes to us as you can see on the image on the right now notice how it can detect multiple objects and multiple classes of objects at the same time this is different from image recognition which understands something might be in an image but it won't tell us where or how many and this is why coco ssd is super useful so let's see in action with a live demo now if we flip to the web page that i've created you can see coco ssd running in the browser i can click on any one of these images like so and we can see we get real-time classifications coming back and notice how it recognizes the dog and the bowl and the cup all different types of objects with high accuracy and the context of where they are in the image but we can go even better than this we can enable the webcam and we can see if it's happening in real time and here i am talking to you live right now today and you can see it's recognizing me as a person with 90 confidence which is pretty damn good now of course what's really good to note here is not only is this working really fast in the web browser with a high frames per second it's able to do this all in the browser on the client side so none of the webcam imagery is being sent to a server for classification all the inferences happening locally on a client's machine and that's really useful for privacy because all their data stays on the client machine which is very top of mind these days okay back to the slides now next up we've got face mesh this model is just three megabytes in size and has the ability to recognize 468 facial landmarks on the human face now not only does this work super robustly we're starting to see real world use cases of people using this in production too on the right hand side we see a demo by muddyface who's part of a l'oreal group for ar makeup try on in the image on the right it should be noted that the lady is not actually wearing any lipstick instead this demo uses tensorflow.js combined with webgl shaders to augment the chosen color onto the person's lips in real time in the web browser and let's see the face mesh in action because it's actually a really cool model so i want to show you it live today as well so over to the demo okay so now we can see the demo and you can see face mesh run running live in the web browser now on the left hand side you can see the machine learning action highlighting on my face where it believes the key points to be but because this is javascript we can go beyond just doing the raw machine learning and we can combine this with 3d graphics such as 3.js to produce this beautiful point cloud on the right hand side as well which you can see me dragging around and rotating in real time at the same time as i'm speaking and you can see how it reacts and updates in much the same way and javascript is very powerful for this kind of stuff it's been designed for the presentation of information from day one and it's very very rich libraries in 3d graphics data visualization and so on and so forth enable you to do this kind of stuff with ease now you can also see here that this is running at good frames per second around 15 in my browser right now whilst i'm live streaming at the same time so do bear that in mind however we can actually flip the back ends to different forms such as web assembly to get more performance on the cpu or we can stay in webgl to use my graphics card to get better acceleration on that so it's up to you what environments you want to execute on okay back to the slides and next up we've got body segmentation this model can distinguish 24 body areas across multiple bodies all in real time now this is really hard to demo live as i need more space in my bedroom but notice from the image on the right how the bodies of each person are correctly segmented with different colors for different body parts now even better we can get the pose estimation to those lines in blue to estimate where the skeleton is so we can do things like gesture recognition and much more in fact with this model we can be super creative as we'll see on the next slide so with a little bit of imagination we can emulate some of the superpowers we are promised from sci-fi movies and i'd like to show you some demos today from both myself and the community to show this in action so first up invisibility this is more advanced than simply replacing backgrounds with a static image for that you wouldn't even need machine learning but notice how when i go into the bed the bed still deforms in the image on the right as i move around or how the laptop screen still plays as they move behind it now this prototype uses body picks which we just spoke about on the previous slides um to calculate where the body is not so it can eventually learn all the background and keep updating parts where it's safe to do so now even better this whole prototype was made in just one day using our pre-made model and runs entirely in the browser meaning for many people you can try it out globally even without having any machine learning background simply click a link and it just works no images are sent to the server for classification leading to real-time results or next what about lasers another member of the community from the usa combined his love for webgl shaders with tensorflow.js to enable him to shoot lasers from his eyes and mouth much like iron man in the movies now this uses the face mesh model that we just previously spoke about to run in real time in the browser without issue and whilst it's a fun demo you can imagine using this for a movie launch to amplify reach with creative experiences for fans and much more or how about teleportation combining tensorflow.js with other emerging web technologies such as webrtc for real-time communication a-frame for web mixed reality or even 3.js for 3d we can now create even more beautiful digital experiences such as teleportation to teleport ourselves anywhere in the world in real time and here you can see me segmenting myself from the bedroom i then transmit my segmentation anywhere in the world and recreate myself in a different physical location with webxr now remember all of this is running in the web browser no app is installed or required to be installed leading to a frictionless experience for the end user and having tried this myself it really feels much more personal than a regular video call as i can walk up to the person hear the audio from the right area in the correct direction and maybe next time i'm presenting to you i'll be able to do so in your own room like this as if i was really standing in front of you and of course there are many other delightful creations we can make too beyond just super powers how about this clothing size estimator that i created here i've created a tool that can estimate your clothing size in under 15 seconds in the web browser to automatically select for you the correct size of clothing on a website now i don't know about you but i can never remember my sizes for clothing and now with this tool i can simply enter my height stand facing the camera and once to the side and it can automatically choose from me the correct size of clothing at checkouts and of course this means less returns and less time wasted overall this was created in just two days and can be potentially be used by anyone with a single click at the point of checkout on a website and even better this is entirely running in web browser so user privacy is preserved as no images are sent to the third party server for classification and one more example from the community here someone has managed to bring an image of a model from a magazine to life using a combination of webs web xr and webgl note even with these fancy particle effects and machine learning running this is actually running on a two-year-old android device and still the performance is great the creativity piece is currently working on making these avatars speak too so that's super exciting and stay tuned for more now the second way you can use tensorflow.js is via transfer learning at some point you're going to outgrow the pre-made models that we've created and you want to use your own custom data and transfer learning allows us to take an existing model use what it's learnt and then apply similar problems of the same domain to this such as recognizing a cat instead of a dog or using the same image recognition model now if you're familiar with machine learning you can of course do this programmatically in code but today i want to show you two easier ways to get started now first up is teachable machine now this is super easy to use and runs entirely in the web browser both for training and for inference which is basically the act of using the model to classify something new now the best way to explain this is with a demo so let's go and try it out okay so now we can see the demo screen and this is basically on the website called teachablemachine.withgoogle.com and if you go to that site you're presented with three options here you can see we've got an image project an audio project or a pose project today we're going to go and try and do a custom image project so let's click on that now we're presented with the following screen we're allowed to add different types of classes on the left hand side so let's go ahead and give them some more meaningful names today i'm going to try and recognize my face so let's put my name here as jason along with a deck of playing cards so let's put the name cards in class 2. and of course if we wanted to recognize more than this we can add more classes by clicking the add class at the bottom but now we need to provide some training data so we click on webcam and allow access and now we get a live preview from the webcam and here i can simply add some recordings of my face at different angles and rotations to give it some variety to understand what adjacent face actually looks like so let's do that right now okay so i've got about 38 images of me moving my head around which is enough for this exercise we're now going to do the same thing for cards and you can see here i've got a deck of playing cards that i'm going to bring to the screen and do a similar thing and it's important to get roughly the same number of images to avoid any bias i've got 42 versus 38 that's close enough so now we click on train model and what's going to happen now is live in the web browser it's going to retrain the top layers to attempt to classify the differences between the training data i presented to it and you can see in just under 30 seconds it's already come back with a live train model and it's currently predicting json on the output here as you can see on the bottom right if i bring the cards into view is his cards and jason cards jason cards and you can see how fast and responsive that is now if this is good enough for what your needs are you can actually click on export model at the top right here and click on download and you can download the resulting tensorflow.js model files that you can then use on any website you wish to then do something useful with so maybe you can control some animation when it spots a cat in your room or something like this or send you an alert or whatever you wanted to do it's completely up to you okay back to the slides now teachable machine is great for prototypes but if you want to launch a production model with gigabytes of training data then maybe cloud automl can be used for this instead it even supports exporting to tensorflow.js in this example we can see someone trying to classify flowers all they've done is uploaded the folders of flowers to google cloud storage and then we can move on to the next step of the training process in the next step we can see how the user can now select if they want to train for higher accuracy or faster prediction times of course there's usually a trade-off between the two so you can select your preference at this point you then simply set your budget and continue to allow the model to train and at the end you'll get the option to download and you can see here once complete you can now export to tensorflow.js as shown simply download the files and host on your website or content delivery network you now may be wondering how hard is it to use this model that you just generated in tensorflow.js well actually it's pretty easy in fact it's so easy if it's onto a single slide let me walk you through this first at the top we've got two html script imports the first one is for the tensorflow.js library and the second one is for the cloud automl library next we have an image tag for a new image that we want to classify in this case i grabbed an image from the internet a daisy but it could be anything it could even be an image from a webcam stream if you want it and then finally we have the actual javascript code but it's actually just three lines of js to do the hard work the first line we simply call await tf.automl dot load image classification and pass to it the location of our machine learning model that we trained in this case it's called model.json and is located in the same directory this is the file we downloaded in the previous step and is simply hosted somewhere on your web server we then use the awake keyword here because the model load is asynchronous meaning that it takes some time to complete this allows us to wait for this to finish before continuing sequentially now once a model is loaded we can then grab a reference to the image we want to classify using document.getelementbyid and pass the id of the element we wish to use in this case it's for daisy image which represents the image tag above as you can see on the top here now finally we can call await model.classify and pass to it the image we want to classify again depending on the model this can take several milliseconds to execute so this always uses your weight keyword too and then you'll get adjacent object return to our predictions constant which we can then loop through and print the results or do something useful with it should also be noted that you can call model.classify as many times as you like with different images once the model is loaded which is how we can achieve webcam detection in real time now finally the third way to use tensorflow.js is to write your own models from a blank canvas now of course to give a tutorial on this would require a whole new talk so today we're going to focus on why you might want to consider doing this in javascript and the benefits you can get if you choose to do so first let's start by explaining the tensorflow.js architecture we've got two apis we have a high level api known as the layers api which is very similar to qrs if you're using python already in fact if you know qrs you'll feel very comfortable with our layers api next we've got a low level api known as the ops api which is more mathematical in nature and allows you to do things like linear algebra and so on and so forth this is similar to the original tensorflow api so let's see how these come together here you can see how our pre-made model sits upon the layers api which itself is sitting above the core or ops api now this lower level api can speak to many different environments such as the client side which includes things like the web browser for example each one of these environments can execute on a number of different back ends for example the cpu which is always available but we've also got webgl for gpu acceleration if it's supported or webassembly or wasm for short if supported for faster performance on cpus and there's a similar story for server side environments via node.js node.js2 note here that our node.js implementation can talk to the same tf cpu and tf gpu bindings that python tensorflow talks to so yes that means you can get the same or better performance as python for model inference with the same cuda and avx supports as you'll see in a few slides time now if you do prefer to use python for your machine learning research you can still continue to do so using qs models and tensorflow safe models in node.js without conversion by loading them directly with our layers and ops api accordingly and this is great as it allows you to integrate with web teams who are highly likely using node.js allowing you to then make your model available to the world with the reach and scale of the web and even better if you want to convert your tensorflow saved model to running the web browser directly on the client side you can use our command line converter to do this as long as the ops are supported by tensorflow.js on our client-side implementation now this will convert the saved model to the json format required so the model on the client side can run in the web browser and let's talk about performance here you can see for mobile net v2 running on gpu and cpu on python and in node.js and as you can see the gpu has a very negative performance difference between python and js in fact i think it's like less than one millisecond there so within margin of error essentially for all intents and purposes this is basically the same result however it gets much more interesting when you attempt to consider more than just the inference times typically an ml model requires pre and post processing code as well and if you convert this code to be in node.js we see much faster performance in node for some architectures as shown on the next slide and here we can see how hugging face who are very popular for natural language processing models managed to convert their distilbert implementation into a full node.js one for the pre and post processing layers this led to a two times speed boost for full end to end results which is just one reason you might want to consider using node.js2 and this is made possible due to the just-in-time compiler that javascript has that optimizes code at runtime and is a unique feature to javascript so now let's talk about client-side superpowers that can actually only be achieved by running in the web browser now the first is privacy inference is performed on the client machine so that means no data is ever sent to third-party servers maintaining data privacy for the end user this is particularly important for medical or the legal industries where it might be a requirement not to transfer the user data to third parties and there's growing concerns around privacy these days so with tensorflow.js you get it for free next up is lower latency as javascript has direct access to the sensors on the device such as the microphone the cam accelerometer and much much more there's no round trip time from the server to the client and back again latency could be close to 100 milliseconds or more if you're using a mobile connection assuming zero latency for processing and inference the maximum fps caps out at 10 frames per second if you're sending images one by one which is less than ideal with tensorflow.js running on device we can go much faster than that of course next up is cost if you're not sending data to the server then of course we can save on costs on the hired cpus gpus and ram that we might be needing otherwise on the server side and assuming you need to file fire up just 10 high memory machines with a gpu you can easily be hitting an additional 60 thousand dollars per year and as there's no server you can just pay for the model hosting and website assets which is far cheaper than running an ml server and then finally here are some benefits to using node.js we can use the tensorflow saved model format as discussed without conversion and of course this is compatible with tensorflow and qrs models we can run larger models than on the client side i believe there's some limitations due to gpu memory limits and so on and so forth that limit the upper bounds of the size of models you can run in the web browser and with node.js we can leverage the full power of a server hardware just like python can it also allows us to code in just one language and this is great for code reuse across the stack and if your existing developers already know javascript which currently 67 of devs choose to use javascript in production according to the stack overflow survey of 2020 that is a great win for you too and node.js itself was the most popular choice by developers in that same survey for frameworks and libraries with over 50 percent of respondents using it and javascript of course is the world's most used language right now there's a huge community and support for this if you choose to do so and then finally we have performance it takes the same c bindings just like python has so you've got the same crude acceleration and avx support on the gpu and cpu respectively and due to the just-in-time compiler in javascript you get that boost if you're doing a lot of pre and post processing too which you wouldn't get in the python land so finally we're going to wrap up with some resources for you to get started if you want to continue your tensorflow.js journey if there's one slide you should bookmark and share with folk let it be this one this slide has all the resources you need to get started our website and api are available on tensorflow.org forward slash js and our models are also available to use on that same website now today we just covered three of them there's actually many more to be used too so do go check that out with fully open source so do look on github and we welcome contributions and for more technical questions check out our google group we've also got lots of boilerplate code showing how easy it is to use our pre-made models in minutes over on copen and glitch.com and finally if you're looking for an all-in-one book deep learning with javascript by manning productions was written by folk on our team and takes you from xero to ml hero in just a few chapters and as long as you know javascript that's all you need to get started so finally a quick shout out to our community do check out the made with tfjs on twitter or linkedin to see many more amazing examples that i couldn't fit into the presentation today new content is coming out every week and it's a great way to get inspired and to learn more about what the community is up to now if you are making something with tensorflow.js yourself please also use the hashtag for a chance to be featured at our future events and blog posts and finally the only question left is what will you make this final example comes from a community member over in tokyo japan by day he's a dancer but he's managed to use tensorflow.js to create his amazing hip hop video with some pretty cool visual effects using body picks now the reason i show you this is that machine learning really now is for everyone and i'm super excited to see how tensorflow.js will enable more people to start their journey with machine learning creatives artists musicians no matter what your background you can still use models in ways you've never even dreamt up by the model creator and as you saw from just a few other demos today we're super excited to see what you create too please do use the made with tfjs hashtag if you choose to do so so we can find your work and with that feel free to stay in touch with me on linkedin or twitter and if you've got any further questions i'm happy to answer them over there thank you for listening [Music] you