Supercharge your web app with Machine Learning and MediaPipe

Key Takeaways

Hi, I'm Jen, developer relations engineer at Google. I'm here to show you how to take your web app to the next level using a custom ML solution from MediaPipe. In this case, it's a custom object detector that detects the presence of three different dogs in photos and videos. But MediaPipe has many more customizable solutions, including image classification, gesture recognition, and text classification. And MediaPipe solutions aren't just for the web. We also offer APIs for Android, Python, and iOS. MediaPipe is adding more solutions in the future, so check out the documentation linked in the description for all the possibilities. Let's get started. To build our object detector, we'll be walking through this code lab. If you'd like to follow along in your own project, then you can do so using the link to the code lab in the video description. So, let's start by walking through this overview. MediaPipe solutions lets you apply machine learning solutions in your apps. It provides a framework that lets you configure pre-built processing pipelines that deliver immediate, engaging, and useful output to users. You can even customize these solutions with Model Maker to update the default models. Object detection is one of several ML vision tasks that MediaPipe solutions offers. MediaPipe tasks is available for Android, Python, web, and iOS. So, you will learn how to incorporate an object detection task in a web app with MediaPipe tasks. And you'll learn this by building a web app that detects the presence of dogs. You can also customize a model to detect a class of objects of your choice with MediaPipe Model Maker. You will train a custom model after adding the code to the web app. To complete this tutorial, you need a CodePen account, a device with a web browser, and a basic knowledge of JavaScript, CSS, and HTML. With that, we can move on to the next step. First, let's check out the starter app on CodePen, linked in the codelab. This app doesn't have the MediaPipe code added yet. CodePen is a social development environment that allows you to write code in the browser and evaluate the results as you build. CodePen isn't required to use MediaPipe. You can use the web development environment of your choice. I'm choosing CodePen here because it streamlines the process of sharing my demo code with you. To make your own copy of the code, click fork. Notice there are three tabs here. One for HTML, one for CSS, and one for JavaScript. I'm going to slide these all over to one side so I can view the JavaScript full screen. All the changes we will make are in the JavaScript section. I click this arrow to open the preview in a tab of its own. What we have here is a basic UI showing two images of dogs. When I scroll down, there's a button to enable the webcam. The cursor indicates that I can click these images, but nothing happens when I do. That's because we haven't added the object detector yet. So, let's get coding. The comments in the code lab have matching comments in the code. That will help guide you on where to add each code snippet. Let's move on to the next step of the code lab. This code uses the Skypack CDN to import the package. For more information on using Skypack with CodePen, check out the blog linked in the description. In your own projects, you can use Node.js with NPM or the package manager or CDN of your choice, such as jsDelivr. You can find details about installing the required package in the MediaPipe guides linked in the description. Under the imports, add this code to declare variables for the object detector and for the running mode. The running mode is a string with a value of image or video, depending on whether you're detecting objects in images or in video, respectively. Okay, we're ready to move on to the next step. Under the initializers, add this code to initialize the object detector object. The file set resolver for vision tasks method specifies the location of the WebAssembly, sometimes called WASM, binary for the task. As the method name suggests, you use it for any MediaPipe vision task for the web, such as image classifier, image embedder, image segmentation, and more. The object detector create from options method instantiates the object detector. You must provide a path to the model used for detection. In this case, I'm starting with this efficient that light model suggested by MediaPipe. Efficient that light was trained with the Coco data set. A large-scale object detection data set that contains 1.5 million object instances and 80 object labels. This model is recommended because it strikes a balance between accuracy and latency. It's both accurate and lightweight enough for many use cases. If you're following along using the code lab, you'll notice that the link shown for the model is different from what I'm using here. You can use either link for this demo. The efficient that what I'm showing is more general, whereas the code lab model shown is customized to identify three dogs. This note links to details about available models, so you can find out more. And I'll walk through model customization after we add all the code to our web app. So you can see how that custom dog detection model was trained. The score threshold is set to 0.3. This means that the model will return results for any object detected with a confidence of 30% or greater. You can adjust the threshold to suit the needs of your application. Notice that running mode is set on initialization of object detector. You can change this and other options as needed later on. Now the object detector task is ready to do some detecting. Let's move on to the next step to see how. In the handle click function, add the code shown here. First, we check to ensure the object detector has initialized. Then, we switch the running mode to image if needed using the object detector set options method. I'll explain the purpose of this step once we add webcam detections. Then, we call object detector detect to run inference on the provided image. This method returns an array of detections, each one with a bounding box, an index of the detected object, a score for how confident the model is, and the name of the label for the detected object. There's a nice example here in the code lab to help you visualize the results. I've created this helper function, display image detections, to handle drawing the bounding boxes in the view. There's kind of a big chunk of code here, but what's important to know is that it iterates through each detection and then displays it. Your object detection needs might involve processing the detections in some other way. Let's go ahead and switch over to the preview tab to check out these results. CodePen automatically refreshes the preview upon saving, but I'm going to refresh again, just in case. Yep, the object detector task accurately detects dogs in these images. If you're using the efficient that model like I am here, you'll notice labels for dogs and maybe even a label for the couch. If you're using the custom dog detection model, then you'll notice that Tika is in the image on the left and Archie and Ben are in the image in the right. If there isn't a bounding box, open Chrome DevTools and then check the console panel for errors or review the previous steps to ensure that you didn't miss anything. The MediaPipe object detection task can also be used on video and live webcam streams. Let's implement webcam detections by following along with the next step of the code lab. In the predict webcam function, add this code. It starts out quite similar to the handle click function. First, we check which running mode is enabled and if need be, switch it. Let's say I clicked on one of the images and ran detection on it. And then I wanted to run detection from my webcam. That's when I would need to switch the running mode option to video. And then if I ran video detection and wanted to click on one of the images to run detection, I would need to change running mode back to image. So, you'd really only need to consider changing this option if your use case involves switching back and forth between video and image object detection. Still, it's good to know how to change the object detector objects options if need be. We get the start time for detections using performance now. We pass this value and the video to the detect for video method. Then we pass that result to our helper function display video detections. Add the code for the display video detections function. It's very similar to the display image detections function. And again, how I'm displaying these results isn't too important, though it can be a starting point for your own projects. You might have a different application for the object detection results. Now, we're ready to test the app. I'm going to go ahead and click save even though auto save is on because I lost more than a couple of essays from not saving back in the day at school. When I refresh my screen, I'm ready to click enable webcam. And here we are. The model detects objects in this image I'm holding up. That's a good way to test webcam detection when you don't have the objects available. The EfficientDet Lite model trained on the Coco dataset has a lot of useful applications. But what if you need a model that detects a specific set of objects? What if, say, I wanted to detect specific dogs in photos and videos? Fortunately, I can create a custom model to use in the object detector task using MediaPipe Model Maker. So, let's dive into customization. I can do this right in my browser using Google Colaboratory, or Colab for short. Colab allows you to write and execute arbitrary Python code through the browser, and is especially well suited to machine learning, data analysis, and education. Yes, my web developer friends, I did say Python, but don't click away just yet. Follow along with this Colab, and you'll find that customizing this model takes only a little code. The Colab is linked in the description. It's also linked at the end of the code lab. And stick around, because we will get back to the JavaScript, I promise. I'll start with a brief overview of Colab, so you know how to navigate. Colab displays code and text in a series of tabs. I can show and hide these tabs using the carrot on the side of the tab. The first tab is an overview of object detector customization with MediaPipe Model Maker. Next, there are some instructions for updating pip and installing the required package, MediaPipe Model Maker. I click the play button to run this code block. Packages are installed in the Colab environment, not on my local machine. Next, I import the required libraries, including the Model Maker object detector API. Then, I download the data set. It consists of 139 images of dogs. This data set uses Pascal VOC format, but you can also use COCO format, a common image classification and object detection format. For Pascal VOC, each image has a corresponding XML file with the bounding box and class name for the object in the box. In this case, the classes are the names of the dogs, Archie, Ben, and Tika. There are several software options available for creating annotations like the ones in this XML file, including open-source options. Now, I load the data from the file. I load training data and validation data using the object detector data set from Pascal VOC folder method. Training data is used to train the model, and validation data is used to test how the model training is going. Next, I set some training options, and I select my model architecture, which is MobileNetV2. The batch size sets the number of samples to work through before the model updates its parameters. For a set of 139 images, a batch size of eight is a good place to start, and it works well for this model. Broadly speaking, the learning rate dictates how big of a step the model makes when updating model weights. The number of epochs is the number of passes taken along the whole data set. The learning rate boundaries parameter is a list of epochs indicating when the learning rate will decay, meaning slow down. Just like when you're getting closer to your destination in a moving vehicle, you slow down so you don't overshoot the destination. The learning rate decay multipliers parameter is a list of learning rate multipliers, which calculate the learning rate at the ith boundary. In simplified terms, this tells the learning rate how much it should slow down. To find out more about available options and their roles in model training, check the docs. I start the training process using the object detector create method passing the training and validation data directories and the training options. This training will take a few minutes. The time it takes your model to train will depend a bit on the parameters you choose for training, but mostly the most notable impact on time to train is the number of data points in your training data. With the 112 images in my training set and 27 in my validation set, this process is pretty quick. Notice that we first have some information about the model layers. This can be helpful as you're getting more involved in the training process. If you're just starting out, you don't need to know what these values mean. During training, we get information about how each training epoch is performing. Performance is measured by loss. That is, how far off the model prediction is from the actual value for an example. There are a few popular functions for computing loss. There are several points on which training is evaluating loss. Total loss, CLS loss, box loss, and model loss. The total loss is a weighted sum of the individual losses. It's a good place to see how the model is performing overall. The CLS loss is classification loss. How accurately the model's predicting the classification of an object. The box loss is how well the model identifies the bounding box of the object. The model loss is essentially the same as the total loss. The numbers might be slightly different, but for our purposes, we don't need to worry about these differences. We also have another set of these losses, but they all start with val. These are the same computed losses, but for the validation data. The validation data is never used to train the model, so these losses gauge how well the model performs with data it's never seen. What you're looking for is that in general, these numbers are going down with each epoch. The numbers may fluctuate up and down slightly, but as long as they're trending down, you're on the right track. If the loss values seem to go more up and down than actually decreasing, then consider a smaller learning rate. If the loss values are decreasing, but don't land in a good place, then consider increasing the learning rate or the number of epochs. Try to only change one parameter at a time and make small changes. Once training completes, I evaluate the model's performance. If needed, I can make changes to my training options to improve the outcomes. Finally, I export the model using the export model method and then download the model to my local machine. Now that the custom model is trained, it's time to see it in action. I can use MediaPipe Studio to try out my model in a prototyping environment. Under the model selections drop-down, I can select choose a model file, and then choose my dog detection model. I've already done that here. I can supply my own images from my computer, which are now listed here. And then, I can adjust these sliders to find the best configuration to suit the web app. It's best to try on a few images, so I can find the best confidence percentage that consistently gives me the results I need. I'll try a couple more, just so we can get the idea. Keep in mind that inference time will vary. Since MediaPipe tasks run on your device, the performance will depend on the attributes of your device. Looking good. I'm ready to upload the model to Google Cloud Storage, so I can access it in my web app. You can store your model in the cloud-based storage solution of your choice. Back in CodePen, I'll change out the link in my object detector. Now in the preview, my web app detects these three dogs. On the left, we have Tica, and on the right, we have Archie and Ben. As for webcam detection, the dogs aren't available in the recording studio today, but I can hold up images of the dogs to my webcam to detect them. Today, I showed you how to add object detection to a web app using MediaPipe Tasks object detection API. I customized the object detection model using MediaPipe Model Maker, and then tried out different configuration options in my browser using MediaPipe Studio. There are more tasks and customization options available. So, head to the MediaPipe documentation, linked in the description, to find out all the possibilities. And if you're interested in getting more hands-on with model training, then why not try TensorFlow.js? It's a JavaScript framework for training models and running inference. You can even train a custom object detection model using JavaScript. You can check out the blog post on object detection, view the documentation, or dive right in with a TensorFlow.js training series on YouTube. All are linked in the description. Now, go out there and build something amazing.