Image classification - ML on Raspberry Pi with MediaPipe Series

Key Takeaways

Hey all, I'm Paul Ruiz, an engineer with the developer relations team on Google ML. And this is the first video in the ML on Raspberry Pi with MediaPipe series, where you will learn about the basics of machine learning, along with how you can use Google's newest on-device machine learning tool, MediaPipe, to add useful features to your own Raspberry Pi apps. This series will aim to introduce some general concepts and vocabulary around machine learning, how to create a simple model, and how to implement some common ML tasks, such as hand gesture recognition, audio classification, and object detection, all on the Raspberry Pi. By the end of this series, you should have enough of a foundation in machine learning with MediaPipe to continue learning and making great apps without getting too far off track with all of the math or underlying details. If you are interested in a lot more of the details behind machine learning, I've also included a link in the video description to our TensorFlow zero to hero video series, which is similar enough to MediaPipe for our purposes. So, let's go ahead and start by talking about what machine learning even is. With traditional programming languages, you provide an input and a function, then the program produces an output. So, for example, you might have a function that adds two to any input number. So, one outputs three, eight outputs 10, and 40 outputs 42. With machine learning, you provide a set of known inputs and a set of known outputs, and then the computer tries to figure out what the function is that connects those two. This process is called training, and it's one of the core steps in machine learning. Without getting too far into the details, training involves solving a set of complex problems, including optimizing models to work on relatively restricted hardware, using specific device hardware for acceleration, and ensuring your models work across multiple platforms, which is why we've created MediaPipe as a way to make these challenges a whole lot easier for our developers. Once training is done and your model is ready to go, your next step is implementing that model on your Raspberry Pi. Though, we'll cover that in a lot more detail both later in this video and in later videos through this series. So, just a few moments ago, I mentioned that MediaPipe is a tool for on-device machine learning. But, what does that even mean? As the name implies, on-device machine learning is a subfield of machine learning and is a way to run machine learning inference rather than training directly on an edge device. This includes mobile phones and tablets, IoT devices, and even web pages. Because of the single attribute, developers gain three specific advantages. First, machine learning is able to run quickly using models that have been specifically designed for edge devices. Next, your apps will be able to use machine learning features regardless of whether or not the device has connectivity. For example, when a user or device is in a tunnel or in a rural area. Lastly, because data is never leaving the device, that data is more secure and private. This has opened the door for a variety of new products and features for developers, especially when combined with all of the options and flexibility that you get with IoT hardware like the Raspberry Pi. While discussing these new ML features throughout this series, I'm going to divide them into four separate categories. Natural language processing or essentially text, which are tasks that you might use to translate text, provide answers from a contextual paragraph, or understand some dialogue. Vision, which involves images or videos to recognize content, track objects, or even provide a modified version of visual inputs. These tend to be the most commonly used features and a few IoT use cases in this domain include barcode scanning, item quality inspection, and object sorting. Audio files that can accept input from a microphone or audio file in order to classify sounds, input spoken language, or check the similarity between two separate audio files. And finally, other for those one-off tasks. These could be tasks for content recommendation, tabular classification, or generative content. This list on the screen is also just a subset of what we either currently support or plan to support with MediaPipe. So, be sure to check out the official documentation at developers.google.com/mediapipe to keep up with the latest as we continue to improve. Great. So, what makes MediaPipe unique in the on-device machine learning ecosystem? MediaPipe handles a lot of the monotonous tasks that go along with machine learning, like flow control, meaning you don't need to worry about your app sending data faster than inference can happen, handling the underlying data structures so that you can work with regular objects instead of a series of matrices, or chaining multiple models together to form even larger jobs. MediaPipe also consist of a selection of solutions, such as MediaPipe Tasks, which lets the developers easily deploy ML features in their apps, MediaPipe Model Maker, which helps you create custom models, and then MediaPipe Studio, which lets you try out tasks directly from a browser without needing to write any code. This video series will primarily focus on MediaPipe Tasks as it's the most relevant to IoT development, though it's worth mentioning that MediaPipe Tasks also works with Android, web, and iOS. I will include links to content focused on MediaPipe Model Maker and MediaPipe Studio in the video description for those of you that are interested in learning more about those two tools. Plus, there will be a video later in the series covering both. All right. Now that you know a little more about on-device machine learning, as well as what MediaPipe is, let's go into something a little more exciting. In this case, we're going to make a simple Python program that performs image classification, and I'm going to use a custom model that can detect a small set of animals. For a bit of background, I have a few chickens in my yard, but one thing to know about chickens is that a lot of other animals would love to eat them. In this example, I've created a new model using a set of images that I pulled from Kaggle, which you can find in this video's description. And I'm using that model to detect things like raccoons, foxes, or cats. If I were using this model in a full project, then I might use that detected information to notify me of anything coming around. That way I can know if I need to fix any parts of my chicken coop, or add any additional fencing. So, before you get started, you will need to set up your Raspberry Pi. The most important thing to know is that you must run a 64-bit operating system on the device. In this case, and throughout this series, I'm going to use the standard 64-bit Raspberry Pi OS on a Raspberry Pi 4B. You'll also need a camera hooked up to your Raspberry Pi for this example, as well as any other computer vision examples. Though you can use either the Pi ribbon camera or a USB webcam. The next thing you will need to do is make sure that you have the MediaPipe dependency installed on your Raspberry Pi. You can do that with this command from the terminal. And since this is on-device machine learning, you are going to need your machine learning model stored on your device. There's a variety of ways that you can do this, including using the wget command from the terminal to get a stock model that has been tested already. Though I'm going to simply place my custom model in the same directory as my Python script. If you do want to use a tested default model, we provide a few options from our developer documentation, which you can find in the video description. Now that the housekeeping is done, it's time to create your new Python script to do the actual classification. While I'm only going to cover the most important parts here, you can find an entire example on GitHub, which I'll also link to below. First, there's a couple imports that you'll need at the top of your program. After you have those, you can create a configuration options object for your image classifier. Some of the available properties for these option objects includes the max number of results that you want to receive from the classifier, the minimum confidence score that must be met before a result is returned, and a callback that will receive the results from image classification. In this case, you will use the live stream running mode because this example is going to use the camera to constantly classify against a video stream. Once you have your configuration object made, you can create the image classifier object that will do the majority of the ML work for you. Next, you'll get an image frame from the camera and do some preprocessing on it, including converting that image to the red, green, blue format that the machine learning model requires, as well as create a new MediaPipe image object that will be used for inference. With that all out of the way, you can call the classify async method on the classifier with that MediaPipe image and a timestamp, which will return the classification result to your callback function. One nice thing about this is that because MediaPipe already handles flow control for you, you don't need to worry about sending camera frames faster than they can be processed. If you do happen to send more data than MediaPipe can infer at the moment, MediaPipe will handle dropping the oldest image to keep your processing time down. Finally, this callback will receive a result object, which you can use to do whatever it is that your goals are with your new Raspberry Pi app. The important part to know is the structure for image classification results that you can see here on the screen. This is what you'll use to navigate through the result object to find the information that you need for your app. In this case, I'm just going to display the results on the screen for the most likely seen item that meets the score threshold that I already specified. Here you can see that I'm able to point the camera at a picture of a raccoon and see that it's classified correctly with a 70 to 80% confidence. We're excited to see all the cool things that you make with MediaPipe on the Raspberry Pi. So, let us know in the comments what you've made or what you want to make, and I'll see you in the next video. Thanks.