Slam Dunk Video Classification Tutorial (w/ TF 2.0 Distributed Training!)

Key Takeaways

[Music] thanks for watching Henry AI Labs this tutorial is gonna show you how to build an image classifier to parse out video data particularly to crop out dunks from a full workout tape this tutorial is powered by the Nvidia data science PC built by digital store this PC has two Titan r-tx GPUs in it then we're going to use with the tensorflow 2.0 distributed training API to scale-up training to the two GPUs and train our image classifier to crop out dunks I think recording the workouts is a great way to stay motivated and to document your progress and overall just make working out more fun so in this example I'm recording my dunk workouts where I put my iPhone on a tripod set it at about the 3-point line and then I just record as I don't so the output of these sessions are I get about 8 to 10 of these uncut videos of about 1 to 2 minutes where most of the video is just nonsense and ideally I just want to see the dunks so for example going from this entire clip just to the four dunks that are contained in the video clip the first piece of code we'll look at is how you go from a movie file like a dot movie file and extract the video frames using open CV - so I'm storing each of the dunk sessions as individual video clips in the raw underscore and then so in order to test generalization I'm gonna be wearing two different shirts and dunk sessions a black shirt and a blue shirt so I have my blue shirt clips and then I have my black shirt dunk clips so now going into OpenCV - I'm gonna open up the folder navigate to this directory the raw black shirt and then I manually put in the clip this probably isn't the best way of doing it but it's just how I did it for the sake of this so then I found that when I read the frame for some reason it's rotated 90 degrees so I used this image utils library to rotate it back upwards and then I just use the I am write function make sure I'm incrementing the counter so I'm not just writing over the same frame over and over again and then I use this to write this into this directory which is going to be the black shirt clip a folder so once I've written the video frames to the folders this is my strategy for labeling the data so basically I have a binary classification problem where I'm either in the frame or I'm out of the frame and so what I want to do is I want to classify the frames where I'm in the frame and then use that to cut out the dumb clips and then avoid all the empty frame nonsense the uncut video so what I did is I go through this the segment of clips and I kind of scroll ahead to find where I first appear in the clip and then okay so something like this so already one interesting characteristic of this problem is that there's a lot of borderline cases like in these frames I'm like kinda in the frame not really in the frame so maybe I would even improve the classification on this if I added a third class to the problem where it's like you know borderline cases - so the classifier can have some more uncertainty and not be overly criticized for it so it actually is remarkable that in the end the classifier gets like 100% accuracy when it's got these kind of noisy frames in the dataset so what I would do is I would go to where I first appear I'd scroll up click this and then I would drag it into the black shirt empty folder and then I would you know move that to the trash and then I would go and get the in Frank folder and I would repeat this up to the clip so data labeling can be a bit tedious but you know once you start to have a sizable data set you can and you have this kind of natural data where it's sort of like contiguously labeled like for example I frame four or five is out of frame frames four oh four 403 etc they're probably out of the frame - so what I can do in the future is I can use the train classifier to help me label the data because it'll be really obvious where the classifier has made mistakes due to this contiguous nature of the natural data so for this project I've been getting the movie files extracting the frames and labeling the data on my laptop and now I want to transfer it over to the Nvidia data science workstation so before I transfer this data I'm gonna resize it down to the 224 by 224 resolution because in this example I'm going to be using the pre-trained ResNet 50 from the Charis applications and it takes as input 224 by 224 and I found a good enough result with that that I don't think there's any need to say implement a customized resident that could take in a higher input resolution so this code all it does is it uses the pillo library I personally prefer using pillow to OpenCV to when I you know when I can so I open the you know you loop through the path in the directory you open it using this kind of syntax then you use the resize function the nearest interpolation from the pillow library you gotta convert to an array I could have just wrapped the numpy right in this but I wanted to be more sighs but you know this code is kind of hacky not really super cleaned up so then I use the image i/o library to write it to the new directory the 2:24 data set and then I increment the count so I'm not writing over the same image over and over again so the 2:24 data set you can see an individual image is is about eight kilobytes compared to if I had the full data set I'd be trying to transfer about three hundred and sixty-four kilobytes per image so we see already a massive reduction in the memory size because as I mentioned if you follow me on Twitter I'm gonna transfer this with a flash drive so I definitely want to cut as well cut away as much memory as possible before I transfer the data so also thank you for all the recommendations I had a look at DBC for data version control and having you know a system for sharing these data sets between machines definitely really interesting stuff to look into in the future so now we've taken the data from the laptop on the flash drive and ported it over to the Nvidia data science workstation and we're going to use the tensorflow 2.0 distributed training API to train this classifier so the first side of this code is to import warnings and filter the warnings to ignore them reviews tensorflow before you know it gives you a lot of warnings and you can get around some of them with this little technique so I've noticed intensive flow 2.0 you need to put attention flow before chaos when you're importing it I haven't looked into it too much and I don't really think it's that you're retaining so I just kept doing it so originally I tried with the image data generator but right now the EOS image data generator at the fit generator function isn't supported with the distribute training API but luckily in this case we don't really need to do too much of this online data augmentation because you know sometimes it's interesting generalization is obviously important but in my case I'm trying to clip these the distribution between my training and testing is is really none nothing at all maybe like a backpack of the basketball in the corner of the frame so I don't even think data augmentation would get me around that anyways so I don't use that data organization in this example so then I messed around with some of these callbacks the reduce learning rate on plateau CSV blogger and early stopping but I didn't really end up using them so import the model obviously use a global average pooling to catch the output of the resident at 50 and then filter it into our new dense layers and then here we go we're importing the ResNet 50 from the Charis applications so first what I did is load the memory a little data into memory using the pillow library so I said this list training X turning Y for the path in the directory it just open it you know converted to an umpire array and then import that one class label to the wine labels and then do this but you know appending 0 to the empty data set so filling up our training in our images and then our labels so then we go through our training data we wrap it into numpy arrays convert it to float 32 s and then we normalize it by dividing it by the max value which in this case is the 255 pixel value and then we convert our labels into numpy arrays as well so now we import our SGD optimizer and this is the part where we're starting to use the new tensor flow 2.0 distributed training api so I kind of just copied this from the documentation if you go to tensorflow 2.0 distributed training api so what you do is you import the distributed strategy the mirror and strategies the only one I think is actually implemented so then you pass it in the number of devices in the case of the Nvidia data science workstation we have two GPUs which is really awesome then we pass it a batch size of 64 to each of the devices and now we're ready to get started so oh we actually don't need to define the step size in this syntax but so we define the strategize scope using this up top now we define our model within this code and then so basically what we do is we take our ResNet 50 then we you get like the output of it like this you take off the last fully connected classification layer because you know you're importing this previous trained model on imagenet classification so it's got this dense fully connected layer at the top which is trying to do a thousand way in this classification but we want to modify that to do binary classification so we're gonna do the global average pooling we're gonna pass it through to dense layers completely arbitrarily chosen I don't even know if you need this or not probably not then we're gonna pass it through the one node activation so a funny thing is originally wasn't getting this work because I accidentally put softmax because I'm used to doing that with the categorical cross entropy but when you're doing the binary across country make sure you have a sigmoid or some other thing on activation of this last layer so then we're just going to compile our model together by having the inputs is the base model dot input and then the outputs is this prediction layer so this syntax is passing in the previous layer is called the functional API in chaos so then we're going define our optimizer and then compile the model and then I'll give you all these warnings that the import warnings warnings filter wasn't able to ignore in order to use the tension flow distributed training API we need to convert it to this tensor flow data object so I kind of just copied this exactly from the documentation I'm not completely sure I mean I think that doing the cash and then shuffling it and then having this batching is you know sort of putting the data onto the GPU and you know some kind of instructions within that that I'm not completely 100% knowledgeable of so the you might want to load the weights to do the training but I decided to just train from scratch so I commented this out so then you just train the distributed training a path this is probably my favorite part about distributed training a guy we've got the Nvidia data science workstation with two GPUs on it in order to scale up to distributed training it's the same syntax as it's always been just model dot fit train data epochs the previous code that we've seen to make it adaptable to distribute a training API really hardly any work at all so we see with the two GPUs the first epoch takes 93 seconds and then the second third 60 365 and this is about 2.5 times faster than doing it the original way in which we're going to look at next so after we train the model for 100 epochs we're gonna save the weights and then we will move the weights back over the laptop for inference and for trimming the videos further encourage the distributed training to potentially 2.0 API and the Nvidia data science workstation well look at the contrary speed if you use just one GPU training this so we see 154 seconds on the first epoch compared to 93 and then 140 344 compared to 63 and 65 so we can already see how we're able to use larger batch sizes when you know we're putting the 64 batch size on each of the GPUs and then we can see how we're getting a much faster training so really interesting this combination of the tensorflow 2.0 distributed training api making it so easy syntax passing tactically to scale up your training to two GPUs and then the built-in setup of the Nvidia data science workstation so now that we have the weights that were trained on the Nvidia data science workstation we're back on the laptop and we're going to load them in into the memory so the way that we do this is we have to import the resident 50 the dense and Cobra cooling layers and the model so we redefine the model again we don't have to do it within the strategy scope because we're not doing distributed training and then we load the weights into this model so now we're going to get into how we use the model to parse the raw video data in order to crop out the frames that actually have me and the frame and actually have the dunks but this definitely isn't the cleanest code you know most beautiful to look at but I'll explain it every line of it and I'm pretty sure it'll be clear how it works so we get the CV to the image utils as we mentioned earlier to rotate the images back up right from how the video capture and cb2 reads the image we got our pillow library numpy and time just because amount of time how long it's going to take just for the sake of understanding it so then we're gonna open up our clip that we want to get the dunks out of we're gonna use the successes just to see if the you know if you've if you've reached the end of the video and then we're gonna use this frame counter in order to label the images within our action frames and we'll get into in the next line of code we're gonna look at why we use this action frames in order to you know clip the videos according to the binary classification labels so we start the timer we open up the video stream so this is just a check at the end to avoid the error where if you successful kind of it'll read this frame and then it won't be zero until it gets to the top of loop so it'll throw an error if you try to like you know rotate this image because it's none type non image but anyways so we rotate the image upwards then we convert it to the pillow image we use the pillow image to resize it because remember our classifier takes in 224 by 224 s input we convert this to an umpire array again a prank a trap to send umpire right but well I am this is well I gonna put dot as type at the end of it but it doesn't matter so then we're gonna normalize it as the normalized divided by 255 is what our classifier is used to so then this is an interesting line of code that you might forget when you first start building image classifiers is that when you're doing inference you need to pass it in you can't just pass it in as the 224 by 224 by 3 you know image array tensor whatever you want to call it you have to put that original so it's like 1 - 24 - 24 3 because this is what your crossfire is used to take your classifier is used to taking in as input batch size height width channel so you need to put just the 1 for the batch size so then we're going to use the model that predict for our label and so as I mentioned earlier there are a lot of these boundary cases where you're like kinda in the frame not really so we're setting a pretty high decision threshold so only if the label is higher than 0.95 do we count this as being in the frame and appended to our action frames list and then we'll increment the frame counter so this section may be where it gets a little confusing so with the action frames basically what we have is we have the locations in the frames where you're in the frame so we're the classifier is predicted one so this is gonna be a list of numbers say frame 405 406 407 up to 587 and then the next number might be 78570 6 say up to 906 and then you know like 1120 a list like this so what this code is doing is its looping through the list and it's coupling them together like this because from frame 405 to frame 577 is a dunk from same 70s from frame 792 to 906 as a dunk etc etc etc so what this code does is it just loops through the action frames and it checks for this kind of gap between 577 to 792 and then it checks whether it's a like an upper bound or a lower bound on the clip so that it can kind of group them this way so I know it's definitely not the prettiest code look at but this is what it's doing it's looping through this list of the 1 labels and then it's grouping them together into the clips that you know look like this so this is saying this frame 150 1517 up to frames 1724 is a don't clip so now that we have our array that tells us where the dumb clips are we're gonna loop back through the video clip and then we're gonna use the lower bound and the upper bound to write the the clips in the video to a separate folder which we're then gonna batch together using the cv2 video writer next so what we do is we first check we're at the lower bound and if so we're gonna make this directory where we're gonna put the clips and then we you know as we continue to loop we'll see if we're in between the lower bound and upper bound you know rotated upwards again kind of silly thing that you have to do and then we'll write it to the folder making sure that we have this counter which we're using to increment make sure we're not writing over the same frame over and over again and then again we're incrementing the clip so each clip will be like clip 1 clip to clip 3 or you can name it dunk 1 dumb 2 and make it less confusing but so then once the counter exceeds the upper bound what we're going to do is we're going to pop it the ductless so using the syntax don't stop pop at index zero compared to pop if you use do dunx top pop it'll pop it off at the end of the list and you want to take it from the front of the list so that you can continue in the syntax so say you get to 578 you would pop this off and then the friend counter is at 579 or whatever and then it will go and so it hits a 792 and then do the same thing again so the code we just looked at writes these image frames into these clip folders so now we're gonna look at some code that I found and I will provide a link in description to the code that I grab from this the only thing is I had to mess around a little bit with this because it's a kind of depending on your operating system I think so using the Mac operating system this is just what works for me and I basically just tried a bunch of things randomly until this work so what we're gonna do is first we're gonna have the 30 frames per second if you want to have it slow-motion you would do say 15 and if you want to have it fast you can do 45 I why you're going to do that for this case but so 30 is kind of like normal speed so what we do is we take the clip folders and then we turn them into movies in the dunks directory so we see that from the clips we can now look at the individual dunks videos that it's made by parsing through the clips kind of stacking them together into a video in the future I'd really like to get this on to the mobile phone check out things like how we can accelerate inference so on one of the trials I had a using the timer the start equals time to time and an N equals two at a time I was timing how long it was taking the model to do inference and was taking about 0.1 seconds so I really like in the feature to experiment with how to speed up that inference how to get this on to a mobile phone and then all the challenges that come with that and also I think would be interesting just to test how this generalizes especially with other dunkers and you know different kinds of gyms so say like in a park or just you know any indoor gym I'm sure would throw this off because just challenge that achieving great generalization can be so thanks for watching this end-to-end tutorial how to get the video frames extract them train an image classifier using the really exciting tensorflow 2.0 distributed training API and the Nvidia data science workstation with two GPUs to really accelerate this training and then how to use this model in order to clip out video frames and you know apply it to some kind of application I hope this may be inspired your own applications and your own ideas for how you can use computer vision thanks for watching and please subscribe to Henry AI labs for more deep learning and AI videos