Realtime Object Detection | Object Detection with TensorFlow | Edureka | Deep Learning Rewind - 2

Key Takeaways

hello everyone first of all I'd like to welcome you for this webinar session where we are performing the real-time object detection using tensorflow so let's get started we are going to understand what is object detection and we are going to have an understanding guess how it actually works when we talk about object detection and in this webinar or at the end of this today's session we are going to perform the object detection so we are going to perform this task of object detection using the Deep learning framework which is tensorflow so first we'll understand what is tensorflow and then we are going to perform the object detection task using one of the pre-trained model that is already available to us inside the tensorflow and we implement it on an image detection or this object detection task okay so that's the agenda that we have for today's session guys so what is object detection the object detection is actually a task that we do and it's a computer technology related to computer vision and image processing which deals with detecting the instances of semantic objects of a certain class such as humans buildings cars in a digital images or videos I know this sounds purely technical but let me make it simpler for you object detection means suppose if I have an object okay suppose if I have a digital image I'll open up an example image for you to make it easier for your intuition let me get an image let's say I'm having this image or this image with me okay when I'm talking about object detection I want to identify what is in this image and I also have to identify where exactly is that object in my image so when I'm talking about object detection I'll be talking in terms of two things so I'll be saying two things I'll be saying what is present in this image I'll also be saying where exactly is the position in this image now in this example photograph I can say that okay what is in this image I can say there is a person's face is present in this image and along with saying as what is present in this given image when I'm doing the object detection I'll also be saying where exactly is that person's face so two things will happen over here one is detection of what and the another is detection of wear and both of these things constitutes the object detection and that's what would happen or that's what we expect from any object detection model we not only have to identify what is present but also we want to detect where exactly is that object is present I'll give you a simple example guys which which is a relevant for today's scenario let's say if I have some uh uh like digital image with me okay I'm having some big Digital Image or a digital image or if I have the similar image with me I'm getting the video feed and I want to detect whether that person is wearing a mask or not the reason because as you know with the current pandemic everyone is expected to wear the mask and follow the social distancing so that everyone will be safe not only the person but also the person around them so I want to build an application where I want to detect whether a person is wearing a mask or not so the task that I want to check is whether that person is wearing a mask or not okay now if I want to check whether this given person or given whether the given video feed or give a given image whether this person is wearing a mask or not as a first step I have to identify whether this given image has a face in it so I have to first detect whether this given image has a face in it and if there is a face I have to detect where exactly is the face in this given image and once I find these two whether I am having a face or and where exactly is that face then I can check whether whether this person is wearing a mask or not I hope you can understand the importance of object detection that I'm talking about so if I want to do any task by looking into the digital images be it uh images or be it video feed I want to identify what is present in that given object or what is exactly present over there in a given Digital Image or the video and along with that I want to identify where exactly is present and that's the Expedition or that's the output that we would get when we perform the object detection okay I hope this gave you some clarity about object detection and what exactly we are trying to achieve when we talk about object detection now with this intuition if you look at the technical terms that is present over in the slides so you can understand that object detection is a computer technology which is related to computer vision now don't get intimidated by the new terms called computer vision guys so this computer vision is actually an ability that we give to the computer to be able to see the outside world now just like we humans see the outside world with our eyes we'll enable the computers to see the outside world how can I enable the computers to see the outside world exactly I can see the or I can enable the computer to see the outside world with the help of camera so object detection is a computer technology related to computer vision where I am enabling my system to see the outside world and once it captures the outside data it could be either video or it could be images so I'll process this individual video or image data and then I'm going to detect that specific instance or that specific image whether it has objects of certain class okay so this object can be something else whether my given image has humans in it or whether my given image has buildings in it or whether my given image has cars in it so all these things we will be detecting now in this example this is the image that I have given as an input and I am detecting what exact where like what are the objects that I am present in this image so in this example I am having a retrieval dog American uh Staffordshire dog and there is one more uh dog I think I am unable to see its complete name so there is another word called another uh variant of the dog and this is one more up so this is one more breed of talk so along with saying what exactly is present in the image I am drawing the bounding box around it which says where exactly is that specific object is present and if you look at the image which I'm having on my left hand side I am doing the same thing I'll just uh start the slides here again so that you can visualize it see I'm identifying what exactly is present on top of my table so I'm checking uh like I I can see there is a soda can I can say there is a hat I can see there is a cup so coffee mug and everything I can see everything now along with saying as what is present in these images I'm also drawing a bounding box around it to say where exactly is that object is present and that's the technology or the that's what we are trying to achieve when we perform the object detection we have we are determining what and also where these are the two things that we'll be doing over here in case of object detection now uh where do we see this application this uh application of object detection one uh common application or one uh application can be is face recognition so I hope everyone has experienced this so if you upload any photograph into your Facebook so it it not only identifies who is present in that image it also identifies like uh where is that person's face is present so it will draw a bounding box around that person and it is going to ask us whether you want to tag that specific person in your image or not so that's an application of wasted that's an application of object detection I'm deciding what is present in the image I'm also saying where exactly is that object and with the current pandemic uh as we know that we want to follow the social distancing so we can we can have the applications where uh we can identify in a large crowd so how many number of people are present in this in this entire crowd and we can also take a step ahead and measure whether they are following the social distancing or not so those are some of the examples for object detection so and yeah another thing can be is like uh for a given image data or for a given video feed so what are the various objects that are present over there now just for a quick fun activity you can like there is a video uh there's a video that I've come across in YouTube guys so which actually talks about uh how uh entire world is actually embracing towards this uh object detection so there is a video I want to share how China tracks everyone so this is one of the video just make sure that you watch it and you watch in your free time and to I understand how much close uh we are going ahead and where exactly we are going ahead in terms of using various object detection tools so just to give you an intuition so in that video they have actually explained uh how China is actually using their various webcam feed and the various surveillance speed and they are trying to uh like they have taken the computer vision to such an extent that they are monitoring each and every individual in each and every place so to do all those things they are actually making use of deep learning models and they are actually perform the object detection over there you once you watch that com entire video you'll understand the powerful things that we can do with object detection guys and another example canvas we can use this computer vision technique to check the industrial quality checking so whenever something is getting manufactured so instead of manually validating everything we can just capture the images and we can have the Deep learning model to run through it to identify how exactly uh like how exactly is it working or whether is it good whether the quality of the product that I'm producing is it a good one or not and in the recent years we are seeing another application of computer vision that is building the self-driving cars now in case of self-driving cars it's a artificial intelligence system which actually observes the outside images so to observe the outside image it will actually have uh cameras around it so it will have the cameras on top fixed on top of it which is which will be a powerful camera and which captures the 360 degree uh video data which are present on the outside and we will have a deep Learning System inside it it get the Deep learning system or this deep learning model gets the video feed from this outside camera it does the pre-processing and once it does the pre-processing it identifies okay if I have a red signal over there in front of me so it is able to identify okay I'm having a red stick red signal I have to stop so that is the prediction that my deep learning model is going to give and with that prediction this uh self-driving car or this system is going to apply the brake so that it stops so that's one of the application of uh using the computer vision guys now even in this scenario we'll be performing the task of object detection because I'll be capturing the outside world and I'll be seeing what is present in that given image and what is the information that it is trying to tell me now with that analysis I'll be taking the action and another example is security so uh like we can use this computer vision application to identify uh whether the person is following the security protocol or not or whether this person is a valid person or not so all these things can be the or are the applications of computer vision and the object detection guys now when I'm talking about computer vision it's the whole uh topic guys like in inside the computer vision we have various tasks so object detection image classification and many things so under that computer vision we have the subtask of object detection so don't get me confused uh when I'm using both the words uh enter like interchangeably so both would mean the same thing so I'm using the computer system to capture the outside world data and give the analysis to me now let's look at the workflow of object detection so if I want to perform any object detection so we actually are going to follow some key steps so some key steps can be is if I want to perform any object detection I'll I need to have a data so I need to have a data I'll use some data with v which is already present suppose if I'm trying to classify or suppose if I'm trying to identify whether my given image has a cat in it and if it is a cat I want to I don't I want to draw a bounding box to represent where exactly is the cat so what we do in such scenarios is as a step one we are going to get the data so the step one is get the data so when I say get the data I'll collect the data from the outside word I can collect the data there are various sources guys I can use my existing business data or I can collect the data from open source so if the open source is not available I can employ some Amazon uh mechanical Turks I can use this uh I can use this service and I can have real people to get the data for me and or I can make use of uh like Amazon image labeling service that they have and I can just AWS so when I say Amazon it's actually AWS guys okay I AWS image labeling technique services that they have and I can use all these Services I and I can get the data now when I'm talking about getting the data what exactly is the type of data are we looking for now suppose the task for me is to for a given image let's say this is a given image this is an image which contains uh tree now let's say for this given image I want to identify what is present in the image and I also have to identify where it is present in the image so if I want to identify both that is what and where then my training data so if I want to have a if I want to detect what is present in the image and where exactly is present in the image then my training data should basically have these two things so if I am building a image classifier or a classifier which can actually detect uh whether my given image has an image of cat or not and if the cat is present where exactly is that cat then my training data so my training data will have the image now along with this image I want to have what is present in the given image so an example can be ascat and I also have to have the bounding box coordinates which says where exactly is that cat is present an example can be something else I can have X comma y comma H comma W which is nothing but suppose if I draw a bounding box I'll choose a different color so X comma y represents the I can let's assume I'm this is the X comma y it represents and I should have uh width and I should have height so my training data that means the data that I that I'll be using it should have the image of that cat or image of the outside object and it should say what is present in that image and it should also specify the bounding box coordinates okay so these are the things that I need to have in my data now once I have the data let's say I'm building some classifier which detects uh whether cat is present similarly I'll be building a classifier whether a dog is present so likewise let's say I I'm building a simple object detection classifier which says whether my given input image has a cat or a dog and if any of this is present it's also going to give me bounding box then I'll be having a set of data which contains images of cat and images of good dog and each and every individual image will have the label which says what is present in that given image and it also says where exactly is present in my given image now I'll be using all this data now using all this data I'll be making use of my deep learning neural network okay I'll make use of my deep learning neural network and using this deep learning neural network I'm going to train this model such that it learns how to it learns to identify what is present in the image and it also uh try it it also learns to identify where exactly the object is present if it identifies what so that's that will be the common flow guys so we'll have the data and we will pass this data to the Deep learning neural network we perform the training and once we perform the training we'll make sure that the Deep learning model that we train so it will have the ability to identify what is present in the image and also it can identify where exactly is that given object or where exactly this what is present in my given image okay now while performing the training so we make use of various uh Network architecture I can make use of CNN or any other models so I'll make use of many Network architecture various Network architecture do the hyper parameter tuning and do all those things and I'll complete the training and once I have the drain data I can send a new image okay I can send a new image and an example can be something like this let's say once I have done the training I'll be sending this data this new image data to my deep learning model and once I send this new image data for my deep learning model my deep learning model can say what is present in the image it is going to say it as cat and to say where exactly it is present in the image it's going to give us the x y h and W that means the bounding box coordinates X comma y and this is the high width and this is the height so that is what my deep learning model is going to give me now as a part of today's session we will be making use of pre-trained model so when I say pre-trained model I am having a deep learning neural network which is already trained so I'll use this already trained model and I'm going to show you as how we can use this already drain model send in the data which is already with us and then we can get the predictions of what are the things that are present in my given input image so that's what we are going to cover in today's session guys now to perform this that is to use this pre-trained model and to send the input data and to generate the predictions we will be making use of tensorflow so tensorflow is the Deep learning framework that we will be using okay and this tensorflow is a deep learning or an open source machine learning platform created by Google and this is one of the go-to tools now to perform any deep learning relevant task previously we were using the pi torch and now we are shifting uh most of our work towards tensorflow because it is backed by the Google and it has very good support for mobile and many such remote devices as well and because of that reason we are seeing most of our applications or deep learning applications getting deployed using this tensorflow open source Library okay so that's how uh we proceed along guys so let's go to the slides and let's walk through the slides now now whenever I want to perform the object detection so I'll have my training data in this example I am trying to understand whether a given image has a car or a bike so I'll have uh images like this and on top of my image so I'll run the uh learning so I'll use I'll like I'll use my deep learning model and I'm going to understand the various features that are present in each Car and Bike so one and this process constitutes the training training of deep learning model and once we have done the training we will have the trained model which can identify whether a given image belongs to either a car or either a bike now once I have trained my deep learning model I'll use the trained retailing model for uh testing it with the new data so let's say this is the new data that I am having which I have not used during my training I'll send this new data I'll send it to my trained model that is this trained model and I'll be able to generate the outcome which is as okay this uh this input image belongs to the class or it belongs to the class of this bike so that's how a typical object detection flow will be so this is actually talking about classification and once we have done that we can also draw a bounding box around it and say where exactly is that given object okay so this is just a overall view as how the training will be now coming to tensorflow as I mentioned already it's the deep learning framework that we have which has been created by Google which we'll be using to create the Deep learning models so this tensorflow has one of the main object type that is called as tensors now when we are working with tensorflow we are going to have all the objects into the data type of tensors and this tensors is actually a standard way of representing the data whenever we are performing the task of deep learning now this tensor is very similar to numpy arrays but the way it is going to give us an advantage compared to numpy array is by making use of tensors in tensorflow it is going to help me in tracking the operations and by tracking the operations I can directly perform the auto differentiation on top of it so that I can make use of those gradients to update the values in my parameters and not only that by making use of tensors we can also get the speed boost because using this tensors we can have the computation run on the GPU rather than the CPU so by running on GPU it has been shown that we are going to get the speed boost of close to 10 times the speed that we could get on CPU and because of those two important reasons we'll be representing all the data in terms of tensors while working with the planning now this tensor as I mentioned it's a similar to numpy array it's a multi-dimensional array and it's an extension of two dimensional tables matrices to data with higher Dimension it's a multi-dimensional array guys suppose if I have a vector I can say that tensor of Dimension 6 which is a single Dimension vector or a single Dimension tensor and if I have a two Dimension tensor I can say this is a tensor of Dimension 6 comma four so six rows and four columns and if I'm having three dimensions so this is a tensor which has three dimensions which is having the shape as six comma four comma 2. okay so this is the basic representation or like this tensors are the objects through which we represent the given data now in tensorflow as I mentioned already so like in tensorflow the computation is approached as a data flow graph so what it actually means is each operation that I'll be performing with the help of my tensor object it's going to track it now by tracking it I can directly go back and perform the back propagation which will directly help me in finding the derivatives for the complex equations because it is automatically tracking by its end so because of that Advantage will be using this tensorflow guys and so what we do in case of tensorflow is we'll have the input data and we'll send it to the tensorflow tensorflow and we'll perform the training on the tensorflow model and once we perform the training we will have the train model with us and once the training has been completed we can just send in the test data and we can validate how exactly are we getting the result so like this is the typical flow that we would follow like we will have the train model with us so we will have the input data we'll use the input data to perform the training and once you perform the training we will have the model trained for us and once the model is trained we can send in the new data and if I'm doing the task of object detection when I send a new data I'm going to get an output that would look like this so it is going to identify what are the objects that are present in the image and along with that it's going to draw a bounding box which says where exactly is that object is present okay so this was about using the tensorflow for performing the object detection so with this understanding let's go to our Google collab notebook and let's implement it in a Hands-On manner guys now here as I've mentioned already we'll be making use of tensorflow so this is going to take care of installing my tensorflow so I'll normally have the tensorflow this is to make sure that I'm having the required version so in my case I I'm actually I want to use the version of tensorflow 2.2 hence I'm just installing that tensorflow 2.2 version for this instance okay which is my Google collab instance so this is going to take care of all the necessary installations and once I have done the installation I'm just creating a folder for me okay so the folder would look like this so I'm importing the OS library and I'm importing my pathlab pathlib library and using both of this uh Library I'm just checking whether I'm having the uh let me do one thing I'm going to restart the runtime so that uh you can understand what exactly is happening over here so this will reset all the progress that I had till now so this is going to assign me a fresh copy of system okay so this is a fresh copy of the system that has been assigned to me and here while make sure that you have switched on the GPU so to switch on the GPU go to edit notebook settings and select the hardware acceleration as GPU and click on save button Now by doing like this you'll have the access to the GPU that is graphical processing unit so I have the fresh Google collab instance that has been assigned to me and this has their Ram of 12.69 GB and the hard disk space of 69 GB and I have already utilized 38 GB of hard disk okay so this is the Google collab instance that has been assigned to me now I'm going to install the required version of tensorflow now as I've mentioned already to perform the task of object detection we'll be making use of pre-trained model now to make use of pre-trained model as a first step we have to create the git we have to get the clone from the GitHub so from this clone of tensorflow models we have to clone it so I'm going to copy that URL and I want to show you over here so this is the tensorflow model Garden guys and if I want to perform the object detection to with one of the with any of the pre-trained model I have to first clone this repository so this code is going to clone the repository from this GitHub link okay so hence I've given us git clone and I have given the models folder so from this models folder it's going to create a clone guys when I say it is going to create a clone it is going to copy everything that is present in this GitHub link okay so you can observe the same in our uh in our left side as well so on our left side we have actually the file explorer currently I'm having only single folder called sample data now I'm going to okay let's wait for the completion of the installation over here so this is taking a while for me to install the tensorflow 2.2 version for me so let's wait okay so it has installed the required version of tensorflow for me now once it has been installed here in this cell I am creating a clone from this GitHub link so I'm going to execute this cell so once I execute this cell so it's going to connect to this GitHub URL and it's going to create the Clone see if you just observe it has created a folder called models and if I expand this folder of models I'll be having various subfolders Community official or bit research and so on see if I open the models I'm having various subfolders Community official or bit research so all these things has been copied over for me that means all these things have been cloned for my working now with this GitHub uh models folder cloned The Next Step would be is we want to make sure that we like all the system requirements will like will satisfy for us to make sure that we can uh work with the object detection API which is our pre-trained model now to check whether everything is installed or whether uh whether we have all the dependencies with us so we just have to run this cell so this cell is going to make sure that it is going to create all the dependencies which are required for us to run and along with that it is going to set up a system so that we can make use of all these pre-trained models so I'm going to execute this cell so please bear with the execution guys so since this is a object detection which is a complex model it would take a while for system to get ready so let's go into uh like verify everything whether every library is present for me and once it verifies everything is present it is going to set up a system so that we can directly run that object detection API now by the time it runs let's explore this GitHub screen that we have over here so Here If You observe uh it says tensorflow model Garden it's a tensor model Garden for tensorflow and it says how you can use this tensorflow model garden and apart from this there is one more link yes there is one more URL which I want to share it with you which is called as object detection tensorflow so I'll add it over here so this URL contains the information of various object detection apis that are available to us inside the tensorflow okay and if you come down it actually talks about what are the individual models that we have inside our tensorflow so all the individual models inside the tensorflow which are available to us is present inside this tensorflow 2 model so so this is the tensorflow to model zoo and if I click on that URL you will be redirected to this URL guys this is that URL and these are the various object detection models that we have and which has been given directly to us by the tensorflow team and all these models that you are seeing so these are the object detection models so these are the object detection models and these models are already pre-trained on Coco 2017 data set if I open the information about this Coco 2017 data set so this is a data set which is actually used for performing the object detection and this has around more than a million of images guys so all this models that we have it has been trained on the on this Coco 2017 data set and it is already pre-trained for us we can just use this pre-trained more trained models and we can do our own prediction that's how easy it is for us to work with tensorflow models you guys okay in our scenario we'll be making use of uh what we call as uh image efficient dnet so I think this is the one efficient debt D5 so this is the pre-trained model that we will be using to perform the object detection in our case okay so here in this cell I am just importing all the necessary helper functions so once I am importing all the necessary functions I'm I've just written a helper function which will help me to create a plot so this is going to create a plot that means it's going to pre-process the image and it will help me to create a plot and along with that I'm just uh creating a label map because as I have already mentioned this this model that we have that is tensorflow to object detection models this has been trained on Coco 2017 data set now to I did to add a name for each and every image that I'll be using I'll have to create a label map hence have created a label map in the form of a python dictionary this is going to take care of it so in this way I'll be creating my category index and it will have its ID as well as its name like this so this will take care of defining the helper function as well as defining the various classes that we have and here this cell is actually going to download the saved model and if it is going to add under this folder of models research object detection test data so I'll open the folder path for you to show you where it will be saved so models research object detection and in this folder I'm going to create a folder called yeah I'll there is a folder called test data and inside this folder we are going to download the file which is a pre-trained model file from this URL so this is going to download the pre-trained model and once I download the pre-trained model I'm going to extract that zip file which is in actually Tower dial.gz format so I'm going to extract it and then I'm going to move everything to this subfolder of this test data guys so I'm going to execute this so this is going to take care of downloading the saved model from the URL that means it's going to download from this tensorflow object detection download from this model zoo and once I have everything I'm going to have it inside my test data see in under the subfolder of test data I have created one more subfolder which is called as efficient debt D5 and which has been trained on Coco data set so we have the pre-trained model with us so what I have already done over here is I have downloaded the pre-trained model for me so this is where we are currently so we have downloaded the pre-trained model now once we have the pre-trained model which is a model which has been already trained we can just send in some test data which is already with us and we can generate the predictions so to generate that here in this uh here like here in this thing yeah like before I do that I'm just loading that model into my tensorflow instance so to load the model I'm just writing as tf.saved model dot load so this is going to make sure that it will load the pre-trained model for me so once I load my pre-trained model I am I'm going to select some test images okay so there are some images which is already available to me inside the subfolder of test images so this is the subfolder of test images and there is an image one image two and image three so if I download any single image so if I download this single image so this is a one image that I'm having with me okay I'm having an image of dog and this is another image that I'm having with me over here so I think this is an image of Beach and I'm having one more image so this is an image of uh various tools that I'm having in this given image okay so what we will can do is we can since we already have the pre-trained model we can just go ahead and load uh load these three images which are present inside our test folder and we can generate the predictions on top of each of this image so I'm going to execute this cell guys so this is going to load each and every individual image and it is going to run the detection on all this input image and it's going to show us as how uh like like it's going to show us as where exactly each and every image is present for us so this would take a while for generating the predictions okay so we have the results for us so there was a typo error guys here I had given us for I in range 3 and normally when we called range function and when we give the value so the input value starts from zero but here I'm having only three images but I was going through four times so hence I was getting the error so make sure that you correct this error if you are working over there just give it as for I in range 2. so this will load every image for us so for the first image it has said this image contains three uh objects so one object e belongs to the person with the confidence of 78 percent and the another object that we have over here is Dog with the confidence of 83 percent and this is another object that is with me which is having the confidence of 83 percent so this is second image it has loaded so it has loaded that uh this is the image that it has and this image has kite with 54 guide with 56 percent guide with uh 89 percent so person with 87 percent so what we have done over here is we have just run through every image that we had over here from our test data test folder and then we have just uh created uh like we have also detected the objects that are present so to summarize what we have done as a part of this today session is we have seen what is object detection so object detection is a task in the field of computer vision and when we talk about this object detection we want to identify what is present in the image and along with that I want to detect or I want to say where exactly is present that specific object that I'm trying to say what okay what and where both are the things that we will be getting when we are doing the task of object detection now to do this in this specific session we have actually made use of pre-trained model which is available to us from the object detection API of tensorflow 2 and one such model is efficient D5 this is the this is the research paper of efficient D5 so we use this pre-trained model and using this pre-trained model we have chosen some of the data which is already present with us inside our folders and we have ran the object detection and once we ran the object detection we were able to see what is present in a given image and along with that we were also able to draw the bounding box to say where exactly is the given object and that's the beauty of object detection and we were able to do all this compound like complex things with very few lines of code and that's the powerful or the That's The Power of tensorflow it will help us in gaining or it will help us in getting the Boost from making use of pre-trained models instead of training everything from scratch so this is about performing the object detection using tensorflow using pre-trained models guys okay so with this we come to the end of today's webinar on performing the object detection using tensorflow guys thank you guys bye