YOLOv3 from Scratch

Key Takeaways

all right i'm pretty excited for this video because in this one we will be implementing yolo v3 completely from scratch so what do you need to be able to follow this video well preferably you have some understanding of how the original yolo algorithm works and i have a completely separate video on that actually and you could just watch the initial 10 minutes of that video if you're completely new to yolo but in this video i will also start with a quick recap of how yolo works and then go and discuss what the main differences between the original yolo and yolo v3 are and then we will implement it from scratch in pytorch alright let's get started [Music] so let's get started and with a quick recap of the original yolo and then the differences between yolo and yolo v3 and lastly we'll implement it but so one thing i forgot to mention is that uh there will also be pre-trained weights of this model that have trained on pascal voc and i'm trying to train it on the ms coco data set as well but it's such a large data set so it's taking a while but when that is done i will upload it to github also all right so a recap of yolo so the idea is in object detection is that we have uh you know object and we want to detect a bounding box for those objects in this case we want to output two bounding boxes right around perhaps like this probably i'll put about a better bounding box than that but when i put a bounding box and you want to say which is the object so in this case it's a dog and you also want to do the same thing for the right right one so without going into too much detail there are many different approaches for solving this problem and the way that yolo solves it um that was sort of revolutionary is that we we we do it in a single forward pass and the way that we do it is that we split the original image into a grid so in this case if we split it into a three by three grid it will look something like this and then the idea is that for each cell we will make a bounding box prediction all right so if we go back then it might be a bit tricky because if we just look at for example the right dog so let's see the right dog right right here we have sort of a three different or six different cells that that the dog is in and similarly for the left dog so how yolo solves this is that we look at the midpoint for each object so if we do that then perhaps the midpoint is somewhere around here and the the idea is then that only those specific cells will be responsible for outputting that there's a dog there so in this case the the sort of the the one in the center um this one will be will be responsible for outputting the right dog and then this to the to the most left will output that there's a dog uh the left dog all right so everything is sort of relative to the cell in yolo so if we zoom zoom in on the leftmost cell then it would look something like this where again everything is relative to the cell so in the top left we have zero zero bottom right we have 1 1. and so if we just imagine that what we want to do now is we want to output a bounding box for this for this object in this one so each bounding box will have four values we will have a x value y value and a width and a height we will also have some some more for the class prediction and so on but we'll get to that in a second just to be able to create a bounding box you need four values and so if we step through one of those at a time the x value will be between 0 and 1 right so i think the x value is across this axis actually and so it will be it will be between 0 and 1 because we have zoomed in on a cell that is responsible for the output so we're sort of assuming that there's an object here so it will be in the in the center uh right here perhaps maybe 0.5 then and similarly for the y value we it will also be between 0 and 1. now for the width this can actually be greater than 1 and that is the case if the object is wider than an entire cell and and so similarly for the height it can also be greater than one if the object is taller than an entire cell so you know if we just zoom out again and look at the entire image we can see that the height of the both of those two dogs are greater than an entire cell so maybe you know maybe this one maybe this one height is maybe 1.5 times one cell height so as i mentioned in addition to the x y width and height each box will also have a so each box will have a prediction i have a probability rather that there exists an object in the cell so if we didn't have that then every single cell would sort of output a bounding box and everything would look super messy so we have to also train the network to say for each cell to be able to say uh you know here's an object take my bounding box or here's not an object don't take this bounding box and addition we also want to know sort of when we output the bounding box we want to know what is in that in that bounding box so in this case it's a dog it's a cat and so on so we're also going to have class predictions and i think we'll we'll get to those in more detail when we actually implement it so that's sort of a quick recap of yolo and again i've made another video where i go into a little bit more depth so you can watch that video if you're still confused over the original yolo so then the interesting part what's new in yolo v3 so sort of the major difference i would say is in the architecture so the original yolo outputted a single uh a single grid of seven by seven what's new and we'll so sort of ignore let's see so sort of ignore everything at the top here for now and let's just focus on these three right here so as i said the original yellow output outputted a single 7x7 grid and what they found was that that was great for um for for large bounding boxes but it wasn't so great for small bounding boxes so like the idea here is i guess let's just try on different scales different uh cell sizes and the the model itself can learn what's useful i guess so for the first one the idea is kind of that here we can now we can we can make it into a a smaller grid size so it's a 13x13 grid size and the idea is that here it can output large bounding boxes so for the middle one we have a lot we have some more cells in the 26 by 26 grid and that is good for sort of medium sized object and in the last one right here we have a lot of different cells so that's great for smaller object so that's sort of the idea is that we will run it through the cnn and now they use a bunch of tricks like resnet skip connections and other tricks right here that are sort of resnet like in that it uses a concatenation of the channels sort of similar to resnet but instead of addition so here to use addition here they use concatenation of the channels so the the uh uses slightly different architecture now but they run it through some com layers and then we get to sort of a scale prediction and we run it through some some common layers right here and we output it output this which is a 13 by 13 uh grid size after that we up sample it using using bi-linear so that's the green one right there we run it through some other column layers and then we do another scale prediction so let me draw so let me write exactly so here here it's 13 by 13. uh here is going to be 26 by 26 and then here it's going to be 52 by 52 and yeah so one thing also is that the original yolo outputted a single bounding box for each cell which sort of uh made it so there could only be one object per cell but now in yellow v3 and i think this was also in yolo v2 we're sort of skipping yolo v2 but in yolo v3 uh we output three different bounding boxes for each cell so a lot more predictions and you know when you take into account that we now use three different scales for our predictions it just leads to a massive amount of bounding box predictions and in fact i think there's uh so in the original seven by seven so that would be 49 uh different in this one we have over one uh over ten thousand uh different bounding boxes so it's a a lot more predictions but it's still uh pretty fast which is kind of what yolo was sort of made famous for i guess was this uh was that it was also very very fast so a new thing that was also in yolo v3 is the introduction introduction of anchor boxes and this was actually also in yolo v2 but again we're skipping that one so yeah anyways so anchor boxes is a way for us to sort of uh encode previous knowledge um into the model so which is you know you can discuss if that's a good thing or not but the idea is that maybe we have one box for vertical object and we have one for horizontal object so instead for the model to come up with a completely new sort of a bounding box we say that you know here you have one anchor box could you slightly adjust this anchor box to fit the object in the image so you can imagine that that's much easier to just adjust an a an anchor box than to than to find a complete a one from scratch so to speak so uh what they did in the original if we go back is that we had so as i mentioned we have three different cells for each so we have three um anchor boxes for each cell for each cell and we have similarly for three different anchor boxes per cell for the for scale two predictions and the scale three predictions so in total we have nine different anchor boxes where the idea is to that the first scale will have larger anchor boxes making it easier to detect larger objects the scale 2 will have sort of medium sized anchor boxes and scale 3 will have the smallest anchor boxes so then you might be like well how do we find those anchor boxes that's sort of a tricky thing to say i mean you wouldn't just do this right you wouldn't just take one vertical and one horizontal so what they did in the yolo v3 paper is that they used a smart way of doing k-means clustering where they used um the loss function was to to sort of maximize the iou um with with all the target bounding boxes so that's um how they did it we're not going to go through that we're just going to take the anchor boxes that he used that he found in the paper all right so we have those anchor boxes rather we know that we have these previous encoded knowledge in the anchor boxes so how do we actually produ produce our predictions so the anchor box is sort of the formula is that we have an output so let's just start with bx that's the x midpoint which is relative to the cell again so we have this which is output from the model then we take a sigmoid of that which means that it's guaranteed to be between zero and one right which is exactly what we want um and then for the y one we have similar thing uh where t y is sort of a just a float value coming out from the model and then we take a sigmoid of that and making it between zero and one for the width uh we have t w which is output float value from the model and then we take the exponent um we take the tw exponent of that value so whenever i see that i'm sort of worried because that sort of taking exponent makes it quite unstable i would say so i'm not exactly sure why you used exponent here but i guess you know he experimentally were able to find that that was made it work better uh but additionally i guess this also makes sure that everything is positive which is a good thing for the bounding boxes so yeah and and the way that we uh encode sort of the anchor boxes so every anchor box has a width and the height so how we produce the the new width is that we take this value right this value and we multiply it with pw which is the uh sort of the encoded knowledge of that particular anchor box width so this is anchor anchor width and then similar thing for the height where we have th coming out from the model we take the exponent of that and then we multiply that with the uh sort of the anchor box height that we have all right so that was a uh sort of a summary of the different the major differences um there might be some small ones uh and i'll mention those as we go through the implementation but anyways let's implement this now to understand all of the details all right so this is going to be split into some different parts uh and i want to start with going through the model implementation then we're going to go through data set loading and then loss function uh train function and yeah that's pretty much it so let's start with the config sorry the model all right so one thing also is later on for the data set and all that everything is going to be in the description if there's anything you need but so for the model anyways we're just going to use to import import torch and import torch.nn as nn so that's pretty much it and uh what i've done also for the model is that i've written down sort of a config file so in the original uh sort of in the paper paper uh they didn't actually mention how exactly the model looks like so yolo v3 was more like a tech report that's what he said in the paper anyways so you had to sort of go through the source code a little bit so what i did is that i went through this entire config file which is written in dark net just not pytorch and i was able to find some patterns and write down a config file of how this architecture looks like so if we go back i'm actually just going to copy paste in that config file so i'm going to paste that in that's how the model looks like all right so it might be a little bit it might be able to you might be able to clean this up even further but this is what sort of the main part is so all right so what i've done here is that pretty much if we have a tuple right so this one for example then this is going to be the out channels the kernel size and the stride all right so every convolution in your v3 use the same padding so we don't have to write up the uh the padding pretty much is it's always going to be three by three kernel size pretty much but so this is the out channels from that comp layer that's the kernel size and then we have the stride so i guess uh in this case we have you know we have three as in channels and then we have 32 out kernel size 3 and stride 2. then we have 64 3 and then 2. all right and then for for a list we have sort of i guess sort of an unnecessary but i wrote b for block and then the number of repeats so uh it used these sort of our residual connections and i guess we can bring out the paper to just watch that so he had some parts written down in the paper so for example here he used these two convolutions right that's what we wrote up in the beginning then he used these residual blocks and this is the number of repeats of those blocks so we had one two eight eight four and this is called dark net 53 and he actually pre-trained this part of the network on imagenet and then he added additional layers for yolo v3 so we'll get to that but anyways that's for the block right here so these are sort of residual blocks where the the thing right there is uh the number of repeats so yeah so this is b and this is the number of repeats so that's it so all of these up to that point is exactly what we just saw that's dark net 53 and then he added additional ones additional layers so we used here this is just out channels 502 kernel one stride one thousand twenty four kernel three stride through stride one and then i wrote s and this is where sort of that branch that we saw so i should bring up this image as well so s right here is actually this right there this sort of this branch now it's not exactly as the image here uh there's only two convolutions here and then it's a scale prediction so the image is a little bit off but anyways that that's what you can imagine where we write s we do the scale prediction and then uh in between we have one com layer then we have an up sampling uh two comp layers and then we have another scale prediction uh one comp layer another up sampling so right this is for the scale one then we up sample to uh to 26 by 26 and then we don't downsample anything because you'll be the same convolution everywhere and so we upsample it to 26 26 then we do another scale prediction right there and then we upsample it again and lastly we do another scale prediction so that's the config file and i'll copy paste in let's see i wrote some information about the config file here so you know tuple is filters kernel size stride every column is the same column lists the structure by b indicating a residual block followed by the number of repeat s is for the scale prediction block and then we also compute the aolo loss on that particular one so that's the branch downwards in the image and then you for up sampling all right so you don't have to understand the detail it's sort of arbitrary right you just need to understand sort of the structure all right so i guess what we can do now is let me just write down the structure of the implementations that we're going to use um sort of the classes so we're going to have a cnn block um which is going to inherit from any module like that and then we'll have pass and we're also going to have this residual block that we talked about right that's for the the list right here so that's for the b so we're going to have that then we're also going to have a class for the scale prediction and that's for s right that's the branch so other than that those are sort of the three essential ones then we're gonna have um i guess yolo v3 so in yolo v3 you know this is where we're actually going to put everything together so we're going to be using the config file in yolo v3 by utilizing these different blocks and scale predictions create the model and then do the forward pass but before we get ahead of ourselves let's just create the com block so we're going to have an init layer and the init layer it's going to take some in channels some out channels uh what else so we're gonna have yeah so one thing we're gonna have also here is bn act is what i called it we're gonna set this to true by default and this is indicating if the block is going to be using by a batch norm and an activation function and then we're going to have quarks as well so we're going to have self.com is and then comp2d of just in channels out channels um and and uh yeah we're going to have we're going to set bias equals to not be an act so if we do use a bn batch norman activation function in the block then we want to then we want to not use a bias right if you use batch norm then bias is a unnecessary parameter so that's why i'm writing not be an act here a little bit confusing perhaps you could probably restructure this and then we're going to send in the quarks which would be the kernel size and uh the the stride yeah all right and i guess the padding as well so then we're going to have self.batch norm batchnorm2d of out channels and then what he used for uh the activation function was a leaky relu so leaky relu of 0.1 in destroyed other than that we'll just do self.use be an activation we'll just set that all right so now in the forward we're going to check if self.use be an uh then we're going to run it just through or sorry then we're going to run it through all of the ones so self.leaky of self.batchnorm of self.com and otherwise uh we're just going to return self.com of x and there are probably some of you that are like why is he doing it that way there are probably better ways to do it um but yeah if you find a better one i'm happy to change it so you know the thing that might be unnecessary is the if statement in the forward pass that might you know we might lose on some performance but i think it's pretty negligible i i would imagine i haven't tested it though so we're then we're going to call the super um just to inherit from n module and that's pretty much it for our cnn block and sort of the reason why i did it with this bn activation is that for our scale prediction right that's going to be in our output then we don't want to use batch normally corellu on our output so in our scale prediction we're going to be using cnn block but we're going to set it to not use batch norman activation and that's what we do right there i guess one thing you could do is you could just create um sort of a com block inside scale prediction and in that way you could ignore the these ugly and if statements and stuff so for the residual block we're going to create our init we're going to send in channels we're going to send in use residual equals true and then num repeats is 1. then we'll call this super of init so the reason here why i'm specifying uh the used residual is because in some cases uh we're going to be using the skip connection and in some cases in just going through the config file uh of yolo v3 he didn't use the residual i don't know why but he didn't so we'll just replicate it as he did it and then for the number of repeats we're going to set it to 1 as default but that is something that's going to be sent in which is indicated by this this right here so that's number of repeats equal to 8. all right so we'll do self.layers it's going to be a module list right and here we're going to store it because we're going to have a lot of repeats so we're going to for for repeat in num repeat so in if um sort of the number of times we are going to repeat it we're going to be adding subset layers a plus equals and we're going to add a cnn block of channels and then how how we did it in the residual block is that he simply divided by channels by two integer by two and then he use a kernel size of one and then the padding should be uh zero which it is by default and then we'll have cnn block again two channels divided by two two channels and then we'll have kernel size three and padding of one so a same convolution so the only thing this really does is it it down samples the number of filters and sort of reduces number of filters and then may uh take brings it back again so that's all that does and then what we'll keep track of as well is if we're going to use the residual which is user residual and self.num repeats is num repeats and we'll get into exactly why we need to store this but yeah i'll just keep it there for now and then in the forward part uh we're gonna send in some image x right so forward of x similarly as we did up here um then we're gonna do four layer in self.layers and we're going to check if we use a residual on this then we want to do x it's going to be a layer of x right so we've run it through this com block remember we didn't change anything right the same padding and we didn't change the number of channels so if we want to use a residual we're just going to add x after we run it through those column layers two column layers but if we don't want to use a residual we're just going to do a layer of x and i guess maybe you can use this on a single line maybe x is layer of x plus x if self dot user israel else layer of x maybe maybe that's a little bit cleaner like that and then in the end after running through all of those we're just going to return x and that is pretty much it for that and i guess we yeah we can just do underscore all right so hopefully you're following so far we've just used a com block and we use a residual block now for the actual scale prediction what we're going to do here is uh we're going to create our init and here we're going to be sending in the in channels and the number of classes that we have so here it's actually where our output is going to be so we first call this super and then uh we'll we'll do self.prediction is nn sequential of cnn block of in channels and then let's see here so i did yeah so how he structured it in the config file is that it took in channel he then multiplied in channels by two with a kernel size of three padding equals one and then for the second one he used cnn block of two times in channels and then he brought it down to num classes plus five times 3 and then b and activation is going to be false kernel size is going to be 1 here so let's see if i did anything correct ok this looks good so the reason why we're doing it this way all right is because so for um we're for each anchor right and we have three anchors for every uh for every single one for every single uh cell we have three uh anchor boxes for every anchor box we need to have one node for each of the classes that we want to predict right that's just normal you're used to that right for imagenet we have a thousand nodes four thousand classes then we have plus five and the plus five here is because we wanna output the bounding box so each bounding box is gonna have a probability let's call it p po maybe so probability that there is an object in that cell then we want to have x y with height okay so that's five values and that's why we get this value right here for the output uh and then we'll do self.num classes equals num classes and in the forward part so in the forward we're going to be sending in x and what we're going to do is we're just going to return self.prediction of x then we want to reshape that that we send in through this last com block what we want to reshape it to is sort of the number of batches now so the number of examples in our batch then we want to do uh three and then self numb classes times five uh no sorry plus five and then x dot shape of two x such shape of three so what we're doing here is that we're just sort of uh making this into two different dimensions instead so instead of having a long vector of all the bounding boxes makes more sense i think to split it so maybe this would be more accurate or make more sense so with three times that and then we're just splitting it right there yeah one thing i also did is that i didn't want let's see i didn't want this to be in that dimension so i also did dot permute so i we want to keep the examples the same want to keep the anchors the same but we don't want to keep this we don't want to keep that it seems we want to put this at the end and the way we do that is we say that this right here should be on this location and then this right here should be in this location so this is 0 1 2 3 right the the third one should be i guess on the second one this is kind of i don't know how to explain this so this is anyways the third one which comes from it should be uh second which is the third dimension and then this one is the fourth one and then the one we want to have at the end is the second one so zero one two we wanna have that at the last position so i guess let's bring an example so we have n examples in our batch each example has three anchors each anchor is going to have let's say we're just doing it for the scale 1 prediction it's going to have a 13 by 13 grid and every one of those 13 by 13 grids is going to have 5 no sorry 5 plus num classes output and if you would change it to scale 2 this would be 26 by 26 if you change it to scale 3 would be 52 by 52 and if you would change the number of anchors that you want to use you would change this to maybe let's say five anchors all right so hopefully you're with me with the scale prediction now what we want to do is we just want to put it all together so we're going to do define init of in channels equals three num classes equals let's put it to 80. so 80 uh is what ms coco used so maybe we don't want to use that as a default let's use pascal voc which is going to be 20 classes starting with calling super and then just specifying number of classes is number classes um self.in channels is going to be equal in channels and what we want to do now is we want to create all the layers so self.layers is self.create.com layers so we want to create that function right we're going to have a forward and the forward we're going to do that later but what we want to do now is create comp layers so here is where we're going to create all the column layers using the config file so pretty much we want to keep track of all our layers in uh not in a list then it won't be tracked by the by uh by pi torch so you you if you're doing model.eval and toggling all of that so you want to use a module list yeah so we want to do in channels is self.in channels now what we want to do is we're going to go through the config so for module in config then we want to check if we scroll up we want to check is this a tuple or is it a list or is it a string right those are the three different types right there so if it's a start if this instance module if it's a tuple then it's pretty simple we just want to do a um a convolution a com block cnn block so we want to take out from that module we're going to have out channels kernel size and the stride from module so we'll take those out and then we'll do layers that append cnn block and we're going to send in the in channels the out channels kernel size is kernel size stride is stride and then we're also going to specify the padding which is going to be 1 if kernel size equals 3 else zero right so that's pretty much it um let's just restructure this a little bit like that so we have layers.append cnn block and that's pretty much it and then we close that layers that append and what we want to do now is we need to set in channels to our channels right so the in channels for the next block is going to be equal to the out channels of this block all right if it's not a tuple we can check uh else if this instance module if it's if the module is a list if that's the case we want to take out the number of repeats which is going to be module of one right that's the second value of the list just to just to be clear here um what we're going through now is the tuple now we're going through the the block right here or the list which has a string which is doesn't tell us anything just to make make make it clear for ourselves but then the second value here is the integer for the number of repeats so scrolling down again we want to do layers dot append residual block of in channels and then num repeats equals num repeats pretty simple so that's it otherwise we want to check and here it will be a little bit more tricky but if it's a string and if the module is equal to s right because we have a scale prediction and we have upsampling but if it is s then we want to plus equals so we want to add when i go through this i'm not exactly sure why i did it this way there's probably a good reason for it maybe it was just to make things more compact why i didn't just include this part in the scale prediction anyways what i did is that i we're gonna when we have um a scale prediction we have sort of the same behavior every time and the behavior that we have is that we use a residual block the same sort of a pattern but we don't use the residual so we set use residual to false and the number of repeats that we do there is one after that we do a com lock within channels and then we take it to in challenge division by two and kernel size is going to be one uh what we do then is then we do the scale prediction taking as input in channels divided by one and we set the num classes equals self.num classes anyways if that's the scale prediction this is the behavior that we have how i got that was just reading the config file seeing when we have an output how does the behavior look just before that anyway so else if the module is an upsample we're just gonna layers that append and an upsample um scale factor is going to be two so we're gonna up sample by two i think you can also set either you want to use nearest or bi-linear i think it's nearest by default but i'm not sure about that after that we're going to set in channels to in channels uh times three and you might be like why are we doing that and the reason for that is because just when we have an upsampling layer that's also exactly where we want to concatenate so bringing up the image right here and again the image is definitely not perfect because uh here uh how i found it at least is that we should concatenate just after the upsample right so this this right here should just be directly after the up sampling layer there shouldn't be any column layers in between here so i that's what i think at least so yeah so that's when we do upsample that's also when we want to uh concatenate and yeah one additional thing is that we want to concatenate the one that was added last so we we sort of add this to a list um where we keep track of the output from this one and keep track of the output from this one and then we get to this point right just after the up sampling we want to take the one that was outputted over there and we want to concatenate it and when we do that we obviously get more channels because we concatenated channels from previously and specifically we have in channels times three all right so this is definitely a little bit messy i think that's just the way it is but i've tried to make it as clean as i possibly can and there are probably some things you can you can do to make it even even better but after all of those stuff we've created all the layers that we want so we're just going to return the layers now we've created that part the only part that's left now and sort of creating the model is one of the most difficult parts but in the forward part now we're gonna keep track of outputs right because we have one output for each scale prediction uh we're also going to keep track of the root connections root connections and those are exactly that i just showed you with the where we concatenate the layers uh sort of concatenate the channels those are sort of the skip connections for concatenation other than that we're just going to do for layer in self.layers and we're going to check first we're going to check if this instance layer is a scale prediction so when it's a scale prediction we want to output dot append player of x but now because we have this branch we want to continue from where we were previously right so we want to do this prediction we want to add it to outputs but then we want to go back so one way i found to do this is if it's a scaled prediction we're gonna add that layer and then we're just gonna continue so we're sort of i guess skipping we're not going to can continue from this place we're going to continue from this place where we were from the beginning so if you scale prediction we'll just add it and then continue otherwise we're going to set x is layer of x and then we'll check if it's a residual block and the layer that num repeats is equal to 8. if that is the case and that's what i found from the config file is that that's where we want to add these roots okay so when if that's the case we want to do root connections dot append x so that that's it after running it through those we want to add it to the root connection all right so if it's not lsip is instance a layer and an upsample then we want to first concatenate x and then root connections minus one and we want to concatenate along dimension one for the channels so that's sort of the behavior that we talked about before is that if we're using the up sample that's that's where we want to concatenate with the last root connection with the last one that we added so what we do then is we just do root connections dot pop so we remove the last one and in the end we're just going to return outputs and that is it um yeah all right so that's the model what we want to do obviously is i'm just going to copy paste in a test right here so if name is main we're going to set the num classes to 20 image size 416 maybe this is a part that i didn't mention so in yolo v1 image size was 448 in yolo v3 it's 416 but it also varies so he used multi-scale training meaning he also used uh it used different sizes essentially but 416 was the uh was the main one so moving on with our with our test file right here we do our model which is yolo v3 of num classes equals num classes we use some random output input as two examples three channels image size image size we do out of modular x and then uh what we're doing here and i guess this is sort of uh a bit confusing perhaps we just want to assert that the first one is of shape you know third two and then three anchors and then 13 by 13 13 by 13 and then num classes plus five but the general uh sort of yeah the general structure or pattern is that we use uh the image size divided by 32 because that's the stride that we've divided by um and yeah so that's the general structure if you want to change the image size so we just want to assert that that those are the shapes and if they are it's a success so let us run this and hopefully we haven't made any errors we probably have yeah so num repeats somewhere that is in the residual so here we want to do for yes range of num repeats all right there's another error yeah all right so self leaky of self bash norm of self.com of x um let's see if it works now no all right yeah so i found i think the error right here is that in the com block here we need to run through both of those yeah so i missed that part this should be an nn sequential um and that should be inside the inund sequential like that so a single layer here is uh has these two com blocks and in that way i think we can uh do the addition but it still doesn't work so yeah all right yeah so i think i found the other error which is that after we do this we want to continue from this block right so the scale prediction is the branch and that's sort of one output and then we're continuing from this branch but here we change the number of channels so we want to set in channels equals in channels divided by two and yeah i mean hopefully this works now but they're sort of uh you you oftentimes get these errors and you sort of just have to debug uh what's what's wrong and now i'm printing all the time as well so yeah let's remove that print statement but yeah as we can see it now works on the test case um and that means that it's probably working uh what you would probably want to do is make sure that it can just over fit well we're gonna do that later on anyways but you would probably want it to overfit some some random noise just to make sure that there's not something major uh wrong in the model but this should be correct now so that was the model it's about let's see 150 lines of code and what we want to move on to now is the data set all right so now that we've done the model we're gonna move on and we're going to do the data set and what i want to show you as well is the structure of the data set so you won't have to do any modification so i'm going to upload all of this to kaggle and there's going to be a link in the description if you want to use pascal voc for example so there's a folder we have the images so 43 000 of them we have the labels for each of those so let me just open one of them so what we have here is the class label for a bounding box and then we have x y with height and so we have perhaps in some images we have multiple bounding boxes so those are all in separate text files and then what i've done is that we have a trained csv and a test csv um yeah which basically says this is the image and this is the text file label associated with that image similarly for the test csv and then i have two additional ones where the purpose of those are just to overfit a single example so eight eight examples and a hundred examples just to make sure that the model works uh the second data set is the ms cocoa which is structured in a very similar way we have images labels test and train csv again all of those will be in the description uploaded on kaggle um so yeah all right so let me just copy paste over the imports that we're going to use so these are the ones that we're going to be using um i guess we can comment out this config because those are going to be all the hype parameters and stuff but we haven't made that file yet so we're going to use numpy os pandas torch pill to load the image uh we're gonna be using uh torch due to uh the data set and data loader uh which i guess we don't need to do here so we might not need that and then we're gonna use uh from utils we're gonna have iou and non-max oppression i don't think we need non-max suppression let's see but anyways the kind of um so those are just basics the the utils file are gonna have some stuff and i'm not going to cover that in depth mostly because i've covered all of those things in separate tutorials like if you would want to know intersection of union i have a separate video in the object detection playlist non-max oppression as well there's mean average precision in that also all of those i have separate videos for this is sort of just focused on yolo and and getting that to work sort of um yeah so i'm going to copy paste that over as well um the utils file so let me just show you so this is going i'm going to copy paste that utils file and if you're coding along then there's going to be a link to github just take the code take the utils file and don't worry about it but if you want to know the details of it you just know that there are separate videos for them all right so what we want to do as well is we got to do image image file dot load truncated files is true this is just so that we don't get any errors when loading the images all right so for the class we're going to do class yolo data set we're going to inherit from data set and did i not did i just remove those maybe i just remove them so from tors.util.data import dataset and data loader so we're going to head from data loader and this class is going to be used both for the ms coco and pascal voc so for the init method we're going to have let's see here so the init we're going to have a csv file we're going to have an image directory we're going to have a label directory we're going to have anchors so right those are the anchors that we're going to use for different scales i haven't broken written them down yet but i'm going to show you later on the image size is going to be 416 as default s are the grid sizes which are going to be 13 26 52 by default classes is going to be 20 and we're going to set transform to none to start with all right so now that we have those we're going to do self dot annotations it's going to be pandas dot read csv of csv file so that's going to be the trained csv or the test csv we're going to do self.image directory oh all right image image directory is image directory and then uh the label directory as well uh the transforms that we're gonna use self.s so let's put as the grid sizes um and then we're going to do self.anchors and we're going to convert that to a tensor so we're going to do towards the tensor of anchors 0 plus anchors of 1 plus anchors of two and the reason why we're doing that is so that we put all the anchors together and what we're putting together are the ones for the different scales right so here for scale the first scale we have three anchors second we have three anchors three anchors so this is for all three scales num anchors is then just going to be self.anchors.shape of zero then the anchors that we have per scale is going to be num anchors so per scale is self.num anchors integer division by three so we're assuming that we have specific set of scales right three scales but you can change the number of anchors that you have self.c equals c and then we're gonna do ignore iou threshold so what they did in the paper is that maybe i should explain this later on but basically what they did in the paper is that um we're gonna set one anchor responsible for each cell in all of the different scales but if there's another anchor that has a high iou also a high iou with the same object and if that iu is greater than 0.5 then we want to ignore those predictions so the idea here is that all right we're going to have for each cell if there's an object we're going to have one anchor responsible for outputting it the one that's responsible is the one that has the highest iou with the ground truth box but perhaps there might be cases where there are several boxes in the same cell that are very good at predicting the same object if the iou is greater than 0.5 then we are going to ignore the prediction for that one okay so we're going to do define length we're going to return length of self.notations all right now for the get item and this is going to be uh yeah so we're going to get index the first thing we're going to do is we're going to get the label path so os path.join of self.label directory and we're going to get from the csv so separate annotations dot i look and then we're going to do index comma one one is because it's at the second column so that's where the uh the text file uh sort of name is uh is located all right so when we have the label path we can load them right and we can do that by using np load txt we can specify the file name which is the label path the limiter is going to be uh space in this case and we also specify the number of dimension is going to be two right we're going to have a row for each bounding box and each bounding box has five values associated with that with it all right so that's it uh the second thing though uh and the thing then we're going to do is to list the only problem here is that we're going to be using uh for augmentation right to augment the bounding boxes and all of that stuff we're going to be using the albumentations library which you can pro if you haven't watched my video on album mutations it's okay but if you want to more in depth on how to use that one you can check out that video but what we want to do for that one is that they want the class label to be last in the bounding box so right the text files right now is class x y with height they want it to be x y with height and then class so how we solve that in a in a in an efficient way is that we use np.roll of that numpy array we're going to specify let's see so np dot roll we're going to do uh four rolls axis equals one so in this way we're just sort of putting it putting all the bounding boxes at the same time on the correct position so that we have classes at the end and that's really all that's going on here so now that we have that we're going to get the image path image path is going to be os path of join of the image directory and then we got to do the same thing of self. annotations of index comma zero so it's just the first column instead then for loading it we're gonna do image dot open of image path and then we're gonna convert it to make sure that it's in rgb i think some are gray scale images in cocoa at least so making sure that it's rgb then for using the implementations we need to also make sure that it's a numpy array so that's all we're going to do then we want to check if we have some transforms uh we're gonna do augmentations is self.transform of image equals image bounding boxes equals bounding boxes and uh yeah that's just lb mutations uh how it how it works so you send in the image you send in the bounding boxes and that way if you rotate the image the bounding boxes are are still going to be uh correct um if you do horizontal flip the bounding box will still be correct you can imagine that doing augmentations by hand or creating the functions yourself takes a lot of work so using a library here is very useful and unfortunately you can't just use torch vision because the only from my understanding the only um sort of uh support for classification so that's why we're using albumentations here all right for the image we take that out from the augmentations the bounding boxes we also take that out from that augmentations all right so now that we all we've done is we've loaded label we've done a transformation which we haven't written yet but if we have a transformation that's what we're going to do for target we're going to do torch.zeros of self.number anchors uh divide integer division by 3 so that we have sort of the same number of anchors at each scale prediction and we're assuming here that we have so we're assuming two things we're assuming that we have three scale predictions right that we have in our model and we're assuming that we have the same number at each scale in our case it's going to be 3 in this value because we have 9 anchors in total then we're going to do s comma s comma 6 and we're going to do that for s in self.s all right so what's the difference here uh why or so what have we done here for each sort of target we have the number of anchors we have three comma the grid size so 13 by 13 for example for the first scale then we have six values and why we need to have six values is we need to have a probability that there is an object so that's the probability of oh i guess probability of an object and that's just going to be 0 or 1 for the target then we're going to have um let's see we're going to have x y with height and class so what the the rest of the values are going to be x y with height uh class so that's in total six values uh yeah so then we have the targets and what we want to do now is we want to go through each of the boxes so for books in bounding boxes um and we want to so this is kind of a tricky thing right we have the targets for all the different scales and we want to go through each of our bounding box in the image and we want to assign uh which anchor should be responsible and which cell should be responsible for all the three different scales right so we should each of the three scales should do a prediction and we need to assign which anchors should be responsible and at which particular cells so how we assign which anchor is responsible is that we check which has the highest iou so for that we need to calculate iou so iou with the anchors is going to be iou we imported that from utils and we're going to send in the box we're just gonna send in the height and the width here uh and we're gonna do self.anchors so this function calculates the iou just from the width and the height so we calculate the ious for the particular box and all of the anchors um right and i guess yeah so those are nine different ones in our case so what we do then is we want to check which anchors were the best so we do iou anchors dot arg sort descending equals true and dimension zero so we're the first one is going to be the best anchor so now we can also take out x y width height and class label from the box right that we loaded right here for boxing bounding boxes and we also want to do now has anchor so so you know we're going to make sure that each different scale i'm sorry that there is an anchor for each of the three scales for each block for each bounding box so we're gonna do false and then times three so just um i guess false false false so this is making sure that each of these three are going to be true at the end and we need to make sure that so that that makes sure that there's a bounding box anchor for each of the three scales that are responsible for predicting the particular object hopefully you're following you can this might be a little bit tricky perhaps all right so what we want to do then is we want to go through for anchor um yeah for anchor index in anchor indus right so we're going through the best one first now what we want to do is we want to so what we want to do first is we need to check which anchor does this um actually belong to right so we have nine different ones there are three uh associated with scale one three associated with scale two and three associated with scale three and those are all different anchor boxes so how we check which scale it belongs to scale index we do anchor index divided by the sub num anchors per scale so perhaps this anchor index might be you know eight then we do divide by three which gives us uh two so it belongs to scale index two which is the last scale so what this tells us this is going to be you know scaling is going to be 0 1 or 2. this is going to tell us which target we need to take out from this list of targets that we have so if it's eight we're going to take out two because we want to take out the last anchor which is associated with the 56 by 56 cell cell grid so when we have that we want to also make sure that we know which anchor on this particular scale are we assigning it to so the anchor on this particular scale that we're looking at is going to be the anchor index and then modulus self. numb anchors per scale and so for example so this is also going to be between a 0 1 or 2 but it's going to be dependent upon which scale we're looking at so here we're computing which scale here we're computing which anchor in that particular scale that we want to use so yeah each of the scales has three different ones which of the three do we want yeah and then we're going to get the grid size so self.s of scale index we're getting the how many cells there are in this particular scale that [Music] we're now looking at and let's see so what we want to compute is um yeah so one thing i forgot to mention is that the labels that we get is the is in in the entire image so let me bring that up again to be clear so the labels that we get here um are associated with the entire image so this tells us it's 0.5 in the entire image okay and this is 0.5 in the entire image so this is pretty much perfectly in the center the midpoint is pretty much perfectly in descent in the center uh in the entire image so i guess perhaps that's the yeah right here that might be the person i think but so that's from the entire image but we want to specify it to a specific cell because remember everything in yolo is relative to the cell so how we do that and we check which i and what which j so which cells are they associated with we're going to do we're going to take int of s times s times y and then into s times x so i is the which um yeah i guess is yeah so this i tells us which y cell and j tells us which x cell so you know if we have uh 0.5 and for let's say x is 0.5 and let's say that s is 13 right then we get that it's in cell 6.5 if we take the integer of that we get cell 6 right so it's in cell six uh in the x coordinate and similarly for the y one so it's gonna be yeah i hope you're following there um yeah so we're just thinking of which cell so the intj then we're going to do which ang um the anchor taken right so we're going to take out that from the scale index and then we want to take out the specific anchor on this scale right so this is checking in this list here of the different target tensors here we're taking out which anchor on that particular scale and then we're taking out the i and the j for the uh particular cells that we're looking at so we're taking out i and j and then we're taking out zero for the probability that there's an object and we're gonna and uh yeah we're just taking out that one because it might be the case that this anchor has already been taken by another object which is you know super rare that you have two same objects with the same bounding box yeah i guess that would be pretty rare but we want to make sure that we haven't taken this before so we need to just check if not anchor taken um and uh that we we don't already have an anchor on this particular scale index for this um for this box yeah so we need to make sure that the anchor has not been taken and that we don't already have an anchor on this particular scale for this bounding box so yeah that's because we want to have three different i want to have a prediction at each of the three scales but if if that's not the case then we want to set targets of scale index then we want to take the anchor on the scale i and j element 0 and we're going to set that to 1. now that we now remember that we want to set things relative to the cell so i and j found which cell are we looking at now we want to find what is the x value in this cell so this is going to be between 0 and 1. and how we do that is we take s times x and we subtract j for yeah so j is associated with the x value so x we take s times x minus j so for example if it's 6.5 we subtract 6 which gives us 0.5 so in in the middle of this particular cell and we'll do pretty much the same thing s times y minus i for the y value so here both are between 0 and 1. okay so the width for the cell and the height of the cell is going to be just the width times s and the height times s right if we have a width of 0.5 for example that means that it's 0.5 of the entire image right so if uh if s is 13 and the width is uh 0.5 then it should be a width of 6.5 so we need to um make this width larger if we um when we're considering that it's relative to the cell so that's just an example but those are sort of the formulas for calculating the width relative to the cell and the height so those can be greater than one and in oftentimes they are greater than one so the box coordinates we're going to set to a torch tensor of x-cell y cell width of cell and the height of the cell all right and then the targets of this particular scale index of this particular anchor on the scale it's you know as you can see it gets pretty messy but on this particular scale on this anchor we want to set i and j and then one to five equal to those box coordinates so one two five here is x y with height and that the last thing to do is the same thing so targets scale index of anchor on scale i j125 is going to be int of class label and we're taking integer here because it automatically loads it as a float but the class label should be an integer all right so that is it for the main part i guess we need to also have an else if statement and we need to make sure that so if the iou of a particular bounding box is greater than the ignore threshold then we want to make sure that there are so we don't punish that anchor box so we sort of ignore it if it has a great enough iou right because we're setting one anchor box for each cell but if there's an another anchor box in the same cell that it also shares a very high iou then we're going to ignore it if it's higher than a particular threshold so we're doing if not anchor taken so we need to make sure that we haven't taken the anchor box right because then we don't want to ignore it and if it's so if it's not taken and the i use anchors for this particular anchor index is greater than steps that ignore iou threshold then we want to set the targets for this particular scale index on this particular anchor i j 0 to negative 1. and negative 1 here means that we're going to ignore this prediction okay in the end we're going to return the image and tuple of targets and yeah so that's pretty much it for the data loading it can look pretty messy i notice now that we didn't use this image size i guess we use it in the in the transformations so we don't need to have that one as you can probably imagine uh the data set loading is actually quite expensive because it's um the bounding boxes are dependent upon the image and we also need to do the conversion of all of them uh dependent upon the augmentation that's been applied and that's why we're doing this every single time and checking which cell is responsible and checking the anchors and all of that so uh that's it for the data set anyways we're going to now move on and do the loss so the loss we're just going to import a torch import torch.nn as an n uh and from utils we're going to import intersection over union okay so we're going to do class yolo loss and we're going to do i guess yolo loss and then module we're going to do init init and super init and what we want to do first is we want to specify all the loss functions that we're going to use so we're going to use mean squared error and then that mean squared error loss and let me see yeah so mean squared error here is for the box predictions then we're going to use b bce e with logic loss so that applies sigmoid as well then for our classes uh in the written in the paper he used bce as well but since we only have one label per box uh we're just going to use entropy so we're going to use cross entropy loss sort of what you normally do right when you do if you train on cypher 10 imagenet you would use cross entropy loss to to to train it and that is the case when you only have one class um yeah so we don't have multi-label loss here but anyways then we're going to do stuff that sigmoid is n and of sigmoid um then we're going to have some constant significant significant signifying there i can say signifying how much each law should be valued so to see so to say so i'm going to have lambda class is going to be 1 we're going to have lambda no object is 10 and there's nothing magic about these values i just wanted something that for a particular threshold to sort of make sense um and in in in this way we value uh no objects more and yeah that's pretty much it then we're going to have lambda object and we're going to put that to 1 and then lambda box we're going to set that to 10 as well so we value box and no object more but honestly i think you can just you can do this and it will still work so there's nothing i don't think there's nothing magical about those and those are not from the original paper those are just for me trying around just testing stuff anyways in the forward we're going to send in the predictions we're going to send in a target and we're going to send in an anchor the anchors so we're going to compute the laws for each of the three different scales right so remember in data set we computed targets for each of the three scales directly and for the loss we're going to compute the laws for a single scale so we need to we're going to call this three times for every forward pass okay so what we want to do first we're going to check where is there an object in this um in this target so we're going to do we're going to take out the sort of binary values where there's an object and where there's not an object and then we're going to check where there's not an object and we're going to set that to zero so why we're doing this is because remember we ignored some predictions if the iu was greater than 0.5 so we just want to make sure that we're ignoring those and we're just taking out the ones that are actually ones or zeros in the target all right so what are the different losses that we're going to have so we're going to have no object loss we're going to have object loss we're going to have box coordinate loss we're going to have class loss and that is it right so we're going to have those two those four so the no object loss is going to be self.bce of the predictions of zero and one of no object and i i'm doing this to just keep the dimensions so that we index with no object we're actually just it's gonna not give us an error so that's why i'm doing that um and then for the target dot dot dot dot means all of them should probably mention that and then we're taking out zero and one this is just going to take out the zero one but when we do um one like that it's going to keep the dimensions so it's not going to remove the last dimension which would give us an error when we're looking for no object which is this binary tensor right there so we're just computing no object loss with bc loss right like that okay so for the object loss uh we need to first get the anchors and no sorry for yeah for the object loss yeah right so what he did for the object loss is not just use once at least not from what i understand so my interpretation of the paper is that um we're going to set it equal to iou so the object score that the cell is going to you know the cell is going to say here's the probability that there's a box in this cell and that that is going to correspond to the iou that it believes the prediction is so for that we need to actually compute sort of the the target at that particular time so we're going to first do anchors dot reshape one three one one two and the reason for this is we want it to match um so this is a little bit tricky i guess but we wanted to match so in the beginning when we send in the anchors they're gonna be uh three by two right so we're sending it in for a specific scale we have three anchors for that particular scale and each anchor has a height in the width and um we then want to reshape it like this uh just to match the dimensions of uh of the of the height and the width that we're going to multiply remember the formulas that we looked at we're going to multiply i guess pw times exponent of w t w is what comes out from the network we're going to take exponent we're going to multiply by p w uh to multiply this for all the examples and all the different scales or sorry for all the anchors and all the different cells right those are that dimension we need to reshape it like this and then we'll utilize broadcasting to do it for all the different cells so the box predictions are going to be torch cat of self.sigmoid of predictions of one two three so we use sigmoid on the height of the x and y coordinates because if we want them to be between zero and 1 and then for the other for the other ones we're going to do towards that exponent of predictions uh 3 5 width and height and then we want to multiply that with anchors and we want to um yeah we want to concatenate along minus one dimension like that the ious we're going to calculate with intersection over union and we're going to and i've covered that exact function in the previous video so box spreads of object and then target of one two five of object so we're computing the box predictions for the ones that um all right so here i just noticed an error we need to make sure that we do towards x of that and then times the anchors and i also noticed one other thing is that we compute the box predictions and we concatenate them and then we want to take out the ones that they're there's just an object in so we would do that that way and then um one thing i did also is i wanted to d dot detach so i don't want there to be some weird uh gradient going sort of uh changing the box prediction dependent on the iou i want the model to yeah to be able to just see the bounding box prediction that it did and then output um uh what the sort of what the iou should be for that particular one um yeah but you i'm not actually entirely sure about this it could probably work even if you remove this dot detach but i felt it was safer to just do that detach make sure that it doesn't impact the computational graph in some weird way uh but yeah anyways so then for the object loss we're going to do self.bce of predictions i'm going to take the o we're going to take out the object so the cells and anchors where there's actually an object dependent upon in the data set which we set to one for the targets we're going to do ious times the target of where there's actually an object so remember all the work that we did here is just to multiply with the ious and that is to say that every object score should not be just set to one if the if it believes it's an object it should be more nuanced and say yeah there's an object here um with a particular so the iou of the particular anchor that particular bounty box that was predicted so it wants to tell us sort of this is what we believe the iou is uh with the true sort of bounding box for that particular object so can be i use times that and hopefully that makes sense that's my interpretation of how they did it in the paper now for the box coordinates uh we what we're going to do is um um we want to do it so i want to do similar as we did here right we want to do self sigmoid of the predictions between 0 and 1 so x and y and we want to do towards x of the predictions times the anchors but what they actually did in the paper just to have better gradient flow is that they did they changed the targets instead of the predictions so they did predictions of one to three and they did self.sigmoid of predictions uh one two three and then so these are the x and y to be between 0 1 for the target what they did is that they just changed the the width and the height by doing sort of the inverse of this so um did it towards that log of the target of three to five and then divide that by anchors so as you can see exponent is the is the sort of the opposite right of uh of this of uh of exponent so log is a log is inverse of exponent and then we divide by anchors one thing i also did is i added one e minus six just to make sure that we didn't have any sort of a weird uh torch log of zero but yeah these this is just for better grade and flow than using this so that's the only thing here and then for the actual box loss we're going to use mean squared error between the predictions of the particular x y width and height and we're going to take out where there's where there is an object and a target of one two five where there is an object right so we just modified the target here uh instead of modifying the predictions to have better gradient flow okay so the class loss is the only thing left and that is going to be a pretty simple it's just going to be predictions of five and forward all the nodes uh from that point where there is an object all right we don't want to predict classes where there's no object and similarly for the um for the target we're gonna do uh five of object and we're gonna do dot long so that's it so i guess what we're doing here is we're just taking out the class label associated with that so you know cross-entropy works that you send in all the logits as the input and you send in which node is the correct one or which class is the correct one and then we want to at the end we want to add all of those together and that's our loss so lambda box times box loss right we're changing the um depending on the constants that we set in the beginning then we do self.lambda object times the object loss then we do self lambda no object times no object loss and then we do self lambda class times uh class loss all right that's the loss function and also i guess um this is just for a particular scale so in the forward part when we when we in our training function that we're going to create we're going to have to um put that sort of add all of those for all the scales but we'll we'll get to that so this is computing the loss for a particular for a single uh scale okay now that we've have we've done the model then the data set we've done the last we've done the most tricky parts we're going to create a config file and the config file i'm going to copy paste in and i'm going to show you what they look what it looks like so the config file is just uh the hyper parameters right so we're specifying which data set that we want to use we're gonna set pascal voc device cuda uh here i've created a uh in a you test i've created a seed everything function if you want deterministic behavior you can i guess we can just remove that for now and we can do uh num workers batch size image size num classes this should be 20 for pascal voc learning rate maybe we can do one e minus four weight decay we have a confidence threshold the confidence threshold is uh where so if the probability of an object is greater than 0.6 then we say that there's an object in that one so that we pick out those ones to calculate the mean average precision we use an iou threshold of 0.5 to remove bounding boxes in non-max suppression we use 0.45 then for s we use um so this is just going to be you know 13 26 52 as we've seen previously but just more general uh pin memory load model save model maybe we can do false on this right now and false maybe then the checkpoint file where it's going to save the uh the model and this is you know just basic pytorch stuff it's in the utils file just save and load model the image directory uh it's going to be the data set plus images that's just the structure that we looked at for the data set here we also have the anchors and so for the first scale we can see that the width and the height are quite large uh for the middle scale we can see that they're smaller and for the last game we can see that they're uh a lot smaller so those are just for different the anchors that was computed with k-means algorithm on the ms cocoa data set so what you would do if you're using a completely different sort of data set is you would probably want to do that but i think those anchors are pretty general and i think they would work on a lot different data sets all right for the for the transforms we're using albumentations and we're doing a dot compose and what we're doing here is instead of doing just a dot resize to 416 uh what they did in yellow v3 is they um kept the aspect ratio of the image and it just padded sort of the the top or the the the sort of the sides of the image with just a constant value so that's what we're doing here and then for we're doing a random crop instead and we're also specifying that the scale should be 1.2 so that the the image here is larger than what we're cropping so we're cropping different parts of the image then we're doing random crop color jitter and then we're also doing we're doing a shift scale rotate we're rotating the image and we're also doing sharing but we're taking one of those two horizontal flip uh blur i'm not really sure what these are to be honest i just put them in there and they looked alright but we're blurring the image we're using this posterize we're sometimes converting it to a gray image we're sometimes shuffling the channels and that's pretty much it and then we're specifying that they are in yolo format and that the mean visibility means that uh basically when we do the augmentation the box uh shouldn't just remove uh rather if we do an augmentation and the object is removed so that the original bounding box is just shown with a 40 ratio of what it was originally then we remove that bounding box yeah and then for the test transform we do pretty much same thing but we remove the augmentation and then we have all the classes of pascal and then we have all the classes of cocoa and that's yeah so that's it for the config file nothing you know we didn't really do anything here just write all the stuff that might impact the model here and then instead of changing all the different functions or different files just changing the config file all right so let's do now the train function right that's the last part and uh you know one thing that you should probably should do as well is you should probably test each of these parts independently to make sure that they work i'm just hoping that i wrote the stuff correctly now which is not a good thing but yeah all right so i'm going to copy paste in the import we're going to use the config that we just created torch optim yolo v3 from the model tqdm for progress bar and then from utils we're going to import minav precision i have a video on that we're importing another function cells to boxes which is just to convert the cells right we're getting an output for each of the cells and it converts it to actual bounding boxes relative to the entire image and that is because we want it in that format format when we're computing you know map non-max suppression and also when we're plotting a couple of examples so and this isn't too difficult uh it's going to be on github but i don't want to go through it because it's a little bit messy but yeah cells to boxes uh git evaluation boxes is just to it's just associated with mean average precision so we want to get all the predicted boxes and all the targets none of this is really essential to yolo this is just sort of uh the steps in between to to evaluate the model and all of that stuff we have saved checkpoint low checkpoint check class accuracy um sometimes we check the accuracy we have another get loaders and that one i might just show you so in the utils file if we go down get loaders all it's going to do it's going to take in the csv path and it's going to import yellow dataset it's going to create a trained data set test data set with the particular config configurations and then create trainloader and test loader and trainee val dataset train eval loader as well so this is just just uh yeah making it not as ugly in the train function but you know this is nothing difficult we're just calling the old data set with the initial values and then doing a train loader with the specific config configuration that we set and uh that's pretty much it so yeah let's go to the train function now and that and uh let's implement this and this right here is just so that we get some performance improvements all right so we're going to do train function uh let's just do the structure right now but the structure is just going to be a main function where we set everything up and then we run the train function and it's going to train one epoch and then we're going to do if name equals main we're going to run the main function so that's it maybe we can do the train function first so we're going to take as input some train loader some model some optimizer um what else some loss function we're also going to take in a scalar and that is because we're going to train it in float 16 so we can have some performance improvements and some um you know essentially double the vram and then we want to have the scaled anchors we want to have we're not going to use a scheduler so let's just do it like that then for we're gonna do uh loop is tqdm of train loader uh we're gonna do leave equals true so this is just to get a progress bar and we're going to do losses we're going to sort of store the mean loss of every epoch so that we can see it in a nice way so what we want to do is want to go through for each batch and the x and y in enumerate of loop all right so then what we want to do is we want to make sure that everything is in the correct device so we do that by doing x.2 config.device right so we have everything in the config file then you do y 0 y 1 y 2 is y 0.2 config.device and so what's going on here is that we have three different scales and we have a label for each of the scales so we're getting out all of the ones and so we're getting y1.2 config.device y2.2 uh y2.2 config.device and that's yeah that's the um making it to cuda then we're going to do torch cuda.amp.autocast and so you know this is just float 16 in pi torch i've also made a video on this um so we're doing it inside this um and it's going to automatically keep track of float sort of what should be in float 16. so we're doing out of model of x then we're computing the loss which is going to be loss function of out of 0 and y 0 and then scaled anchors of zero so when we're sending in the scaled anchors uh we're doing that because they're gonna going to be a dependent on the on the cell size that you use for the image right so the in the config i wrote down anchors right here that are copy pasted from the paper but these are uh in the entire image so the width and height dependent on the entire image we want to have them as dependent on the specific cell so we want to scale them approac appropriately dependent upon the cell size so we just do it um we're going to do it in the main function but just to no let you know that's the scale of anchors then we want to uh we want to add so let's see we want to do like that then we want to do the same thing pretty much but we want to do it for y so out of one y of one scale anchors of one and then plus loss function of out of two y two scaling curse of two and that's it then all right so then we want to append this to losses so we're going to append lost.item we're going to do optimizer.0grad dot scale of loss dot backward then scalar dot step of optimizer and also scalar dot update so that's you know pretty similar to loss backward optimizer step you just have to do these instead and then we want to update the progress bar so we're going to compute the mean loss which is the sum of the losses divided by the length of the losses and then we're going to loop.set postfix where loss is equal to mean loss so in the progress bar we're keeping track of the mean loss all the time and that's it so that's what we're doing in the train function now for the actual main function how we're going to get our model which is yolo v3 the one we did we're going to set num classes equals config.nom classes and then we're going to do dot2 config.device the optimizer is just going to be optim.adam of model.parameters parameters learn rate is going to be config.learningrate uh weight decay is going to be config.weight decay um and then we're going to get the trainloader test loader and then we're going to get train eval loader of get loaders so this is just a train loader with augmentation test is just test data set with no augmentation train eval loader is just a train data set with uh no augmentation so this is just if you want to make sure that it's not over fitting all of that stuff but we're not going to use that now so i could remove it i guess but we could keep it there it doesn't matter too much so train csv is config.dataset plus uh train dot csv and then test csv path is config.dataset plus train uh test.csv and that's it now we want to check also if config.loadmodel uh then we're just gonna load checkpoint uh of config.checkpoint file model optimizer and then we're gonna send in the learning rate as well so okay so for the scaled anchors now we're going to do them so that there's a there are tensors so towards that tensor of config.anchors so we're just making all the anchors to tensor then we want to scale them so we're scaling them towards that tensor of config.s and then we're doing unsqueeze one unsqueeze two and then dot repeat one three two so the reason why we're doing this is just so that we can do the multiplication so yeah we're unsqueezing it to make sure that it has the same number of dimensions as the anchors and then we're multiplying so all the anchors in scale 1 should be multiplied with that particular value of of s 0. so for example we have three anchors maybe i should go to the config file in the anchors here all of these are for scale 1. we mult we want to multiply all of these with the value of s that is particular for that scale and similarly for this and similarly for for this and to make it possible to multiply all at the same time we're just doing it this right here and then at the end we want to make sure that it's on the correct device so dot to config.device and that is it so now we're just going to set up the training part so we're going to do for epoc in range of config.num epocs and we're going to call the train function of the train loader of the model of the optimizer of the locks function the scalar and the scaled uh anchors maybe i missed something here so scaled anchors and that's it but we haven't specified the loss function so the loss function is just yolo loss and then the scalar is going to be torch dot cuda dot amp dot grad scalar so we're sending those in right there and we're also going to send in the scaled anchors okay so we're running it one epoch this is one epoch then we're doing uh if config.save model we're doing save checkpoint of model optimizer and that's it and then maybe we can also do um so i'm going to copy paste this in because this is not really yo this is just a utils file so all we're doing at every tenth epoch we're running check class class accuracy on the test loader we're also running get evaluation boxes so we get prediction boxes true boxes and then we're running the mean average precision so we're just checking uh what's the map value on um every tenth epoch you know you could check out those specific ones but that's not this is just to make sure that our implementation is actually correct and uh to evaluate our model okay we've written a lot of code like so much code so let's just take let's just take eight examples and then yeah we will take eight examples for the train and the test function and maybe also maybe we'll just train on the test loader because we don't want to maybe it will be too difficult with all the augmentation so we'll just use the test loader and make sure that we can overfit on that with minimal augmentation and in the config file we can also you know remove the weight decay so no regularization and yeah i'm expecting to get errors now so let's run it and let's see all the mistakes i've made all right so we need to go back to n-dim which is right there uh in the data set file yeah this should be nd nd uh min so the number of minimum dimensions is going to be two so even if we just have a single bounding box it should just still be two dimensions right let's see the next one all right so this is in the loss function so we're doing something wrong with no object loss right so we need to do parentheses right there and we need to do parenthesis right there then we need to remove that one right this is the input of the predictions and these are the targets all right let's see the next one all right so i think the next one here is that we're taking out the objects right where there's actually an object and we need to do the same thing for the target so we're just going to take out the ones that there's actually an object in all right so i've debugged it quite a bit and one thing is that let's change the number of examples to 100 i think it still works for eight but using 100 we can make sure that it actually works and sometimes i've encountered issues with batch norm using um very low you know using a single batch so let's use 100 and then in the data set one large mistake here is that we're resetting this to be class label but we should only be sort of the the fifth uh index should be class label right otherwise we over writing otherwise box coordinates will be overwritten with the class label and then the another mistake is that after we've taken this box we need to make sure that we've actually set it to be taken for this scale index so you know we're looking at an anchor from for a specific scale and after we've taken an anchor we want to do has anchor of scale index equals to be true so that's another thing and in the config file i've changed the learning rate to the party constant of course 3 minus 4 and that's pretty much it i think i also changed the confidence threshold to 0.4 all right so now let's run it and make sure that this actually works all right so i ran it for this is i guess 20 epochs and we can see that we got class accuracy 70 99 27 map got to 0.5 and then after two additional ones we got 99 percent map and i i stopped it there um but so uh yeah i'm going to release a pre-trained weight on pascal voc that has about 78 map on test and ms coco i'm currently training but it but it has around 42 map so it still has a a bit to go one thing that i think could also raise the accuracy which we didn't implement is multi-scale training that he did in the the yolo vt yola v2 and yellow v3 paper meaning we're training on different scales so maybe we could change this 416 to 608 and and other image sizes so that's one thing you can definitely do to improve it um but yeah that's it for yolo v3 from scratch um i'm gonna upload everything to github and it's gonna be in the description but thank you so much for watching the video and i hope to see you in the next one [Music] you