DETR PyTorch Implementation | DETR Tutorial Part 2

Key Takeaways

This is the second part of data video which covers the implementation. The entire video I have broken down in four sections. We first quickly do a recap of everything covered in data part one. So all the components that we would have in our implementation matching predictions to targets computing loss everything. We then go over the complete PyTorch implementation for building data and training it. post that look at results that the authors provide in the paper as well as what we get by training this implementation on VOCC data set and finally do the same for visualizations to get better understanding of what the model is learning. So let's first start with the recap and obviously if you have not seen the data explanation video it's better to watch that first. This recap is just meant to refresh highle aspects of deta. In data, we first go through a pre-trained backbone which gives us a feature map containing semantic information about different regions of image. This gets converted to a sequence of grid cell representations which then gets passed to a transformer encoder. The encoder is layers of self attention and MLP and generates a representation of the image features at all spatial locations within the transformer. To bake in positional information, we use sinosoidal position embeddings, but add it at all the attention layers to the Q and K inputs. rather than only once at the input layer of encoder. The last layer of encoder returns the representation of the image features at all spatial locations. The decoder of data is then going to use these image features as well as n object queries. This decoder would also be layers of self attention and MLP but additionally also have cross attention layers. Now these queries are all initialized as zeros. But in order to have the model produce different outputs for each of these queries, n embeddings are used inside decoder in the self attention layers. These n embeddings which the authors call output positional encodings. These are added at each self attention layer similar to how spatial position encoding is added at the attention layer in encoder. For self attention, these embeddings are added to the Q and K inputs. For cross attention, the image features that are returned by the encoder are used as inputs for K and V and object queries as inputs for Q. Similar to self attention, these N embeddings are added to the Q inputs and the spatial position encodings are added to the K inputs. Using all the layers, the decoder generates representations of these object queries. Then finally, each of these object query representations are used to make single bounding box prediction and also single set of class probabilities using two MLPS. For each object query, deta predicts the cx, c y, width and height coordinate of a box. These coordinates are normalized from 0 to 1. So w and h equals 1 means the predicted box is of same dimension as that of image. for training these sets of predictions where each prediction has classification score and box coordinates. These predictions are matched to the set of target objects. This matching happens such that the assignments are unique that is each target is assigned to just one prediction and each prediction also is just assigned to one target. The assignments are also such that the cost of assignment is minimum. We use Hungarian matching to give us this optimal unique assignment. But in order to actually use the matching, we need to define the cost of each assignment. So the cost of a target and prediction assignment is a combination of classification and localization cost. The classification cost is negative of probability prediction for target class for the predicted box. This is because we want to have high cost assignment if the model predicts a low probability for the target class. Localization is L1 distance between predicted and target box coordinates. But in addition to L1 distance, the authors also have negative of generalized IOU which will ensure that assignments where target and predicted boxes have a very high overlap, those assignments have a very low cost. The total cost is a weighted sum of these individual costs. After we have defined this cost and computed it for all target prediction pairs, we'll use Hungarian matching to get the optimal assignment. The assigned predicted boxes are considered as foreground with the assigned ground truth boxes label considered as their target label and the unassigned ones are considered as background. Using this optimal assignment, we train the model with a combination of classification and localization loss. Classification loss concerns with all the boxes and is cross entropy between ground truth label and predicted class probabilities for each box. Localization loss which concerns only with the foreground boxes will be smooth L1 loss between predicted and target box coordinates. There'll also be another component of localization loss which will be generalized IOU loss which aims to increase the overlap between target and predicted box through the course of training. Similar to cost, loss is also a weighted sum of these three components. We also saw in the part one video that the authors rather than computing loss just from the predictions of final decoder layer, they use auxiliary losses as well. We actually take all decoder layer outputs for each layer. Get predicted boxes for each object query using the same shared MLPS. then run matching separately and formulate total loss as a sum of losses from all layers. This aspect of using all decoder layers output is only for training. For inference, we'll just use the last layers output and return predictions made by the MLPS with the last layers object query representations fed as input. That's it for our recap. And with that done, let's start going over the implementation. Now this implementation is not exactly like the official one, but all the building blocks and configuration parameters are same. Wherever the implementation deviates from the official one, I'll make sure to call it out in the video. Okay, first let's start with the main data module class. This module will have the responsibility of first computing the box predictions given an input image by calling all the necessary components. Then during training it will compute loss based on targets and predicted boxes. Whereas during inference it will just return those predictions. As we have seen in part one, data model is going to have a backbone call then a projection from channels returned in feature map by backbone to the dimension of transformer layers. We'll then have the feature map projection go through encoder layers take the encoder output and using that and query objects go through the decoder layers. The outputs of decoder are then fed to shared MLPS. One for predicting classification probabilities for each query object and the other for predicting box coordinates. Just a point to remember that our model is going to be predicting boxes in cx cy width and height format. So there would be instances where we would have to convert the model output from cx cy width and height to x1 y1 x2 y2 format. For initialization of data class, we require three inputs. Model config which will have all the values for necessary parameters to build our model. number of classes in our data set and the class index for background class. We first get everything from config. Backbone channels is channels returned by the layers of backbone. Although in paper the authors use ResNet 50 or ResNet 101 but for quicker training I have used ResNet 34. So the channels here will be 512. We will be making a lot of such modifications where the motivation is really just to have higher training speed because data takes a large number of epochs to converge. But now because of these modifications we will also ultimately have lesser detection accuracy. Dodel is our hidden dimension of transformer which by default will be 256. The next parameter is number of queries. For VOCC data set, I have kept this one as 25. The official implementation uses 100 for Coco data set. But in VC, there are very handful of images that have more than 25 ground truth objects. I think that number is 15. So in our data set class, we'll filter those out and work with this smaller number of query objects. Number of classes for PC is 21 including background. This is the number of decoder layers. I've kept it as four as opposed to the official implementation using six layers. And same will also be the number of encoder layers. While computing the cost for assignment, we had different weights for each component. So these are those three weights. Then we have background class weight. So this is to ensure that our model does not give too much or too less importance to the background class for classification loss. I've used the same value used by the authors for cocoa data set which was 0.1. But ideally we should play around with different values as the optimal value will depend upon things like how many queries we have, how many objects are there on an average image of our data set and so on. Later we'll see that the weight of all other classes during cross entropy loss computation is going to be one whereas that of background class will be this 0.1 value. NMS is only used for visualization and not for evaluation. So the next one is just the IOU threshold for NMS. The index for background class is zero. However, the implementation will also work if it's the last index, but it should be one of those two. These were all our parameters. Now we'll initialize the backbone, the projection convolution layer, transformer encoder and decoder and prediction MLPS. Starting with backbone. The backbone used by me like I mentioned is ResNet 34. And similar to the official implementation, I have chosen the batchnom parameters. So neither will the assigned parameters of batchnom be updated during training nor will the running stats of mean and variance by default. Again for speed up I will be freezing the entire backbone as opposed to training it with a smaller learning rate which the authors do. This is our projection convolution which will take the 512dimensional feature map returned by ResNet 34 and project it to 256 channels which is the value of DMO used. Next we initialize our transformer encoder. Let's look at the encoder initialization method as well. Our encoder is going to have four layers. Each encoder layer is going to have a self attention. So these are the four self attention modules for the four encoder layer. After self attention we initialize the MLPS for all layers with relu activation that is the activation used by default by the authors. Then we initialize normalization for attention and normalization for MLP layers followed by dropouts for both. The authors have both pre and postnorm variants of transformer in their implementation where in postnorm the normalization is after the residual connection. But here I'll only implement the pre-normalization version. In case of pre-normalization, the authors after the final encoder layer also have an additional layer norm. So this output norm is exactly that the layer norm for the outputs of the final encoder layer. With the encoder initialization done, let's go back to our data initialization. After encoder, we'll initialize the n embeddings for object queries. Remember these embeddings will only be added at the attention layer. The inputs to the decoder will be the object queries which we will initialize later in the forward method of data and it will be initialized as zeros. By the way, for initializing these embeddings, you can also use nn.bedings. Next, we initialize our decoder. The initialization of decoder looks similar to that of encoder. We work with four layers and initialize everything for all the four layers. Each layer also has the same modules as encoder. Just that now we also have cross attention, its normalization and its dropout. When using auxiliary loss, each decoder layer's output prior to being fed to class and bounding box MLP under goes through normalization. For that, the authors use a shared layer norm for all the decoder layers. This output norm is that shared layer norm. That's it for our decoder initialization. Now, let's go back to data class. The last two layers that we'll initialize will be our prediction MLPS. Classification MLP will predict logits for all 21 classes of VC data set from all the object query representations from all decoder layers. All decoder layers because of the auxiliary loss during training. Bounding box MLP will be predicting CX, CY width and height from all the object query representations from all decoder layers. The bounding box MLP has two hidden layers with relu activation. This is same as what the authors use. So now we have initialized all layers of data backbone backbone projection query embeddings encoder and decoder as well as prediction MLPS let's now go through the forward method of data whether we are in training mode or inference mode we'll always have to call all the layers and get predictions then in training we'll compute loss and in inference postprocess detection. S we'll first go over the common logic and then look at training specific code followed by inference code. The input to the forward method is X which is a batch of images of shape batch size cross 3 cross H cross W. For our training I have fixed the images to be always of 640 cross 640 dimension. This is different from the official implementation which uses multiscale training. The next input is targets which the data set class will return. These will be a list of dictionary where the if element is a dictionary with labels and boxes as keys and it contains the ground truth target information for the if image in the batch. The common logic for training and inference would be going through all layers and computing predictions. So let's go through that part. We first have the backbone call from which we'll get the 512 channel feature map and our spatial dimension will be 20 along height and width. Why 20? Because our input is always 640 + 640 and stride factor is 32 for ResNet 34. So 640 by 32 gives us 20. We project this 512 cross 20 cross 20 feature map from 512 channels to Dmodel channels. So conv is bat size cross dmodel cross 20 cross 20. Then we initialize our fixed spatial position embeddings. The implementation of this is same as the one that I used in diffusion transformer video. In case you have not seen that video, then we are first getting height values of all spatial locations. Then width values for all spatial locations. Creating a grid of these height and width values. Get demodled by two-dimensional sinosoidal position embeddings for all the height values. Do the same for all width values. And now that we have position embeddings representing both coordinate positions for each spatial location, we simply concat them to get demodelled dimensional position encoding for each spatial position. This is slightly different from the official implementation in data repo especially on how the sign and cosine components are used to get the height and width embeddings and the authors also do some scaling and normalization. I will link the code in description so you can check out the exact difference but I have just reused the dit implementation. Now back to our forward method of data. After calling this get spatial position embedding method we get our demodel dimensional position embeddings for all spatial locations. And since our feature map was 20 + 20 and default dmodel is 256. So that means this is of shape 400 cross 256. Now our image features which is this con route up till now it was of batch size cross dmodel cross 20 cross 20 shape but prior to calling the encoder we have to convert it to a sequence. So this is what we do here. We reshape our image features to be batch size cross number of spatial locations. So 20 * 20 cross dodel. So basically collapsing the spatial dimensions into a sequence. With this sequence of image features as well as our spatial position embedding, we call the encoder. Let's look inside the forward method of encoder. In this method, we are simply going to go through all the encoder layers and for each layer call all the modules that we initialized. So for each layer first normalize then self attention dropout and then residual connection. We have seen that in data the authors rather than adding position embedding once at the input layer they add it at all the attention layers. So for self attention we'll add the spatial position embeddings to query and key arguments of the self attention call. For the first encoder layer the input which is this in underscore attention will just be the sequence of image features. And for later layers it will be the output of previous encoder layer. After self attentions residual connection we have normalization MLP dropout and again residual connection. Finally after all the layers we normalize the last layers encoder output and return it. I also return the attention weights but that is just for visualization. Going back to det's forward method, this encoder call will return the output of final layer and it will be of shape bat size cross number of spatial locations cross dodel. Next, we'll call the decoder which needs the encoder output as well as query objects. We already have the encoder output. But we have not yet initialized query objects. So let's do that. We do that by initializing this all zero tensor of shape batch size cross number of queries cross D model. I've kept number of queries as 25. So this is batch size cross 25 cross 256. with this zero initialized query objects encoder output n randomly initialized query embeddings and spatial position embeddings for the encoder image features with these we call our decoder. The forward method of decoder is what we'll look at next. Now since we will be using decoder outputs from all layers for auxiliary loss, we are going to save each layer's output in this list. Then for each decoder layer, we'll call all the modules. First normalization, self attention, dropout and then residual. And within self attention now we'll add the query embeddings to the Q and K arguments just like we added spatial position encodings in self attention of encoder. Then we have normalization cross attention dropout and residual connection. In cross attention the query objects will attend to encoder returned spatial features. So for QKV attention now Q will be the query objects but with query embeddings added. K will be the encoder features but with spatial position encoding added and V will just be the encoder features. After cross attention we have normalization MLP dropout and restwell. Then we append this layer's output to our decoder_outputs list for auxilary loss computation but only after they are normalized via the shared layer norm. This decoder outputs which is a list of four values since we have four layers is what the decoder returns. Now back to the forward of data module. This decoder call will return our representation of query objects for all decoder layers. So it will be of shape number of decoder layers cross batch size cross number of queries cross d model. For our case four cross batch size cross 25 cross 256. Then we pass this query object representation to classification and bounding box MLP. Classification MLP will give us the classification probabilities for each of the query object for all decoder layers. And similarly, bounding box MLP will give us the box coordinates for each query object for all decoder layers. This sigmoid here ensures that our outputs are normalized from 0 to 1 which is also how the target is going to be normalized in our data set class. So cx cy being 0.5 means the center of predicted box is same as the center of image. Now we are done with all the common logic for training and inference. Next we'll do things differently for both of them. Let's go through training first. For training, this is what we will do. For each decoder layer, we will first compute the cost of assignment between each prediction and target box. We'll then hand off these costs to a library method that given the costs returns to us the optimal assignment and then we'll compute loss based on those assignments. All this is what happens inside this loop. So let's go inside this. We first get the class and box predictions for this decoder layer. Next, we'll be computing cost and getting optimal assignments based on this layer's prediction. Now, since our matching algorithm is not really a continuous one as a small change in the assignment cost might lead to a completely different optimal assignment. So no smooth gradients. And this is why the authors perform the entire matching within this no_grad mode. So it's like we are handing off the current predictions of the model to the assignment solver and whatever assignment it gives us we'll use that to compute loss and the model will be updated to make those selected assigned predictions closer to their assigned targets. This entire part is serving the same purpose as using IOU to match anchor boxes to ground truth targets and then computing loss based on those matches. The only thing that we are changing is our strategy of matching. Now moving to the actual assignment logic, we first get the classification output for this decoder layer and reshape it to aggregate all the predictions for the entire batch together. So the first dimension will be batch size times number of queries and second will be the class logits. We also have a softmax to move from logits to class probabilities. Similar reshape is also done for bounding boxes. Why are we doing this? The reason is that we'll compute a cost matrix between predictions made for the entire batch. So batch size times number of queries and targets for the entire batch. So all the ground truth boxes for the entire batch. And then for each batch image, we'll segregate the relevant cost matrix portion and use our library method to get the matched assignment for that batch image. Now readability wise, the code would be much simpler if you compute matching cost in a loop for each batch item separately. But the official implementation does it this way. So I thought best to use that itself. That way, even if you're using the official implementation, you will end up with a clear understanding of what's happening behind those lines of code. With that goal of computing cost for entire batch items together, we concatenate all target labels and boxes. So, the target boxes shape will be number of ground truth objects in entire batch cross 4. At this point, it's better to take a specific instance of a batch with hypothetical target objects. So let's say we have a batch size of two. The first image has 10 targets and the second image has five ground truth targets. Our prediction boxes aggregated for the entire batch would be 50 + 4 50 because 25 is our number of queries. So we predict 25 boxes for each batch image and our batch size is two. Our target boxes would be 15 + 4 as first image has 10 targets and second has five. Now let's start computing costs for these pairs. Our classification cost would be negative of ground truth classes probability. So class prob is of shape batch size times number of queries cross number of classes. So here what are we doing? For each predicted box of the batch we are getting the classification cost for all targets in the batch. This would make our classification cost matrix to be of shape batch size times number of queries cross number of targets for the entire batch. So 50 + 15 in our example. This means that each row I will have the classification costs of assigning the IAT prediction box in the batch to all the target boxes in the batch. This would obviously have some meaningless cost entries like cost of assigning predicted boxes of first batch image to a target box of say second batch image. But that's fine. Later we'll get rid of those entries before calling the Hungarian matching algorithm. Next we compute one part of localization cost which is L1 distance between all predicted boxes and all target boxes. But before that we also have to convert predicted boxes from cx cy width and height format to x1 y1 x2 y2 format as that is how our targets will be defined later in the data set class. So we need to have both predictions and targets in the same format before computing distance. The L1 localization matrix would also be 50 + 15 same as the classification cost as we are computing cost of assigning any predicted box in the batch to any target in the batch. The second part of localization cost is negative of generalized IOU between all predicted and target boxes in the batch. Now each of these costs have different weights. The default value that the authors use is 1, five and two as the weights of classification, L1 and generalized IOU cost. So we use these three cost weights to compute our total cost. As of now, what do we have? We have the aggregated cost matrix. So now we are ready to do the assignment for each batch item. All we have to do is extract the relevant parts of cost matrix for each batch item. To do that, we need to have an idea of how many ground truth boxes are in each batch image. Because we already know that we have 25 predictions for each batch. So if we also know the target numbers for each batch image, then we can extract our relevant portions. Let's see how that is done. We first reshape the cost matrix. So our cost matrix as of now was 2 * 25 + 15. After this reshape it will be 2 + 25 + 15. Number of targets per image will be a list containing number of ground truth targets for each batch item. So for our example this will be a list of two. The first element having value as 10 as the first image has 10 targets and second element having value as five. Then we split the cost matrix using the last dimension with these number of targets. Our total cost per batch image now becomes a list of two. The first element being 2 + 25 + 10 and the second one being a cost matrix of shape 2 + 25 + 5. Now we are ready to call our library method. This match index variable will be holding the list of assignments for each batch image that the Hungarian algorithm is going to return. In this loop, we go over each batch item and for each call this linear sum assignment which is the library method that we are going to use to solve the assignment problem for us. So total cost per batch will be a list of two elements. First one being 2 + 25 + 10 and the second one 2 + 25 + 5. Accessing the zero index will give us cost of shape 2 + 25 + 10. So the first element and then from that accessing the first element of first dimension gives us 25 + 10 shaped cost matrix. This is nothing but the cost of assigning 25 prediction boxes of first batch item to the 10 target boxes of first batch item. So this is how we are extracting our relevant portion of the cost matrix for each batch item. Similarly, when batch index is one, we first get 2 + 25 + 5 and then accessing the second element of first dimension gives us our 25 + 5 matrix. So now for all the batch images after we have extracted the relevant cost matrix portion linear sum assignment performs the matching for us and gives us a sequence of matched prediction box indexes and sequence of matched target box indexes. We append that to match index list. Regarding the format of what linear sum assignment returns, it basically returns to us two sequences. The first sequence is the list of matched prediction box indexes and the second sequence is the list of matched target box indexes and both are in the same order. So the first element of both are matched to each other. Similarly, second element of both are matched together and so on. But obviously we won't always have as many targets as prediction boxes. So this list will only contain prediction box indexes that ended up being matched. So with 25 predictions and for our example of 10 targets for the first batch image, only 10 predictions will be a part of these match index list. Remaining 15 will be unassigned. The target for which needs to be background class. Once this loop is done, then we have our optimal assignments. Now we'll use this matched index to compute loss. Our predictions be it classification output or bounding box output of this layer. For both the first two dimensions are batch size cross number of queries. So 2 + 25. We need to be able to extract matched prediction boxes from these since for classification we have to set their target labels correctly. And for localization, we only want to work with foreground boxes. So we get the corresponding values for the first and second dimensions for the matched prediction boxes. The first dimension values will give us the batch index 0 or one in our example that the matched prediction box belongs to. And second dimension will give us which of the 25 query slots does the matched prediction box belong to. This part extracts the batch indexes for the matched prediction boxes. Remember match index will be a list with two elements. So this loop will run for two iterations. When value of i is zero, prediction index will be the sequence of matched prediction box indexes from first batch. Which means when we create our tensor with all ones and then multiply it by zero essentially we are creating a tensor with as many zeros as there are matched prediction boxes in first batch image. Similarly we repeat for second batch image and then concatenate those together. So after this loop is done batch indexes will be a sequence of 15 values. In our example, first 10 will be zero and next five will be one, which basically gives us the batch indexes for all the matched 15 prediction boxes for the entire batch. Next, we'll get the query indexes of the matched boxes. So again, the first 10 values will be the indexes out of the 25 query predictions which ended up being matched for the first batch item. The next five will be those indexes which got matched to one of the five targets of the second batch image. We have the batch and query indexes of the matched prediction boxes. Now we are going to extract ground truth class labels for these matched prediction boxes. And how do we do that? We go over the match index list and this time getting the sequence of target box indexes for the same batch index. We then get the labels for the ground truth boxes at those indexes. Do the same for all batch indexes and then concat them together. Here also we'll get a sequence of 15 values which are nothing but the target class labels for the 15 matched prediction boxes. Now that we have the matched prediction boxes and their target class labels, it's time to compute classification loss. For that purpose, we first create this target classes variable which is going to hold the ground truth classification targets for all prediction boxes. This will be of shape batch size cross number of queries and initially the target label for all predictions is set as background. Then we get those prediction boxes that are assigned to some valid targets. So in our example, the PR batch indexes will be 15 batch index values and these will be 15 query index values. We select these indexes from the target which are basically going to give us the index positions for prediction boxes that got matched for them. We set their label as the 15 target class labels that we extracted from before. So basically we are setting the target of assigned prediction boxes to labels of their respective match target boxes and the remaining boxes will continue to have background as their target class as that was the initial value of the target classes variable. This part here is just for ensuring background class weight for loss computation. So every other class's weight is one and background classes weight is whatever is present in the config. Again this is something that I have not really played around with. So I just use the same value as the authors used for Koko data set which was 0.1 but like I mentioned before it should ideally be tuned for VOC. Using the target classes and class logits predicted for this decoder layer. we compute the classification loss. Then we move to computing the two parts of localization loss. For localization loss, we are only concerned with prediction boxes that got matched. So we get those first. Then we move to getting the target boxes. So from matched indexes of each batch, get the target boxes. And after doing the same for all batch indexes, concat those boxes together. Going back to our example, this will be a sequence of 15 target box coordinates. So 15 + 4. Now we have 15 matched predicted box coordinates and 15 ground truth box coordinates. So we convert predictions to x1 y1 x2 y2 format and use loss. This conversion is again because our target boxes are in that format. After that we compute the generalized IOU loss using these matched predictions and target boxes. Now we have computed all the loss components. So after that we multiply these losses by their weight coefficients and we are done. But we are done only for one decoder layer. All this that we did matching and loss computation was just for one decoder layer. So once we loop over all decoder layers we'll get a list of classification loss values and bounding box loss values and we'll update this data output dictionary with these losses. This data output dictionary will be used in our training script for the backward pass in the forward method of data. This is all that is needed for training part. Let's move to the inference logic. For inference, we don't need to do any matching or loss computation. After getting predictions from the model, we can straight away jump to this line. For inference, we'll only use the last layers output. And after softmax, we'll get the classification probabilities for all the 25 query predictions for all classes. For AP optimization, the authors always return the best foreground class, even if that is not the class with the highest prediction. So, as an example, for a particular box, let's say it has the background score of 0.4 and some other foreground classes score is 39. Then for AP computation, we won't discard this box just because the highest scoring class is background. We'll use the 39 foreground class prediction for this box instead. To do that only, we get the max class label apart from background class here. We then convert our box predictions to X1 Y1 and X2 Y2 format as my AP evaluation script also expects that. Then the official implementation just returns these boxes. And this is what makes DTOR so elegant. We don't require NMS or any kind of filtering prior to getting predictions. However, in my implementation, I have also additionally added a low score filtering and NMS, but it's only used for visualization. For AP evaluation, I don't use either. But for visualization, I use a score threshold of.5 and also NMS with.5 threshold. Once we have done filtering for all batch items, we return the list of detections for the entire batch. The attention weights again are just for visualization. Okay, so this was everything inside data. Let's also quickly look at the data set class and training script to complete the implementation walk through. The data set class for VOC is same as what we had in SSD video using the exact same augmentations. The official repo of DTOR uses random resize and crop as the augmentations and their data loader paths all batch images with zeros such that all images end up having the same dimension. But for ease of implementation, I've instead resized everything to same 640 cross 640 size and just added a bunch of different augmentations. Our background class is at index zero. And then in this method, we get all images and the target boxes and labels for those images. Most of it is same as VC data set class of previous videos. However, for dattor specifically, I ignore images which have more than 25 ground truth targets which allows me to use 25 as my number of queries. In get item, we just get the target boxes and labels for the image and additionally normalize the box coordinates to be from 0 to 1. But the box coordinates are in x1 y1 x2 y2 format itself and not cxc cy wh format which is what our data model is going to be predicting in. And this is why in our forward method we did all those conversions. The training and inference script also is same as pass detection videos. We get the config and let's just look at the config also once before proceeding with training. Data set params has training and testing set. So if we wish we can either train on VC 2007 train val images or 2007 + 2012 data set number of classes is 21 and background class index is zero. The default image size for both training and testing is 640 cross 640. These are all the model parameters which we used in data class and all of which I've already talked about in detail earlier. The training params is our batch size, learning rate, epochs and all other stuff. Like the official implementation, I also train for 300 epochs with a learning rate of 1 e minus 4 for the first 200 and then 1 e minus 5 for the next 100 epochs. As I mentioned earlier, for evaluation, neither do we have score filtering nor NMS. But for visualizations, we do both. Back to our training script. Using the data set config, we instantiate the training data set. Then instantiate model using the model config and resume from a trained checkpoint if present. Since I always freeze the backbone, I can just use one common learning rate for all trainable layers. But if you also wish to train the backbone, you can just separate the backbone and non-backbone parameters and use different learning rates for them. The weight deck used by authors was 1 minus 4. So I also just use the same value in the training loop. After a forward pass, we get the loss values for classification and localization for all layers. Then we sum them up to get our total loss. After that, we do a backward pass using this loss and update the model based on gradients. The inference script I'm not going to go over as it's same from previous videos. There is infer function which allows to visualize predictions on an image. an evaluation method which computes AP for VOCC test data set. In both we'll call our model with the images and get detections from the data output dictionary that the model's forward method returns. I have NMS and low score threshold as arguments in the forward call of data. And again for evaluation we don't use NMS or the score threshold but for visualizations we do use both in the forward call. Then the model gives us the detections and we first visualize the image with ground truth targets and then next visualize the image with predicted boxes. So this was all the code for implementing and training data. Now let's look at the results of our model training on VOCC data set as well as results that the authors mention in the paper and even go through few visualizations. The authors compare the performance of data with faster RCNN baseline on Coco data set. They also compare with a faster RCNN version that is trained longer as well as with more augmentations and uses generalized IOU. In terms of AP, DTOR gives comparable performance to that of faster RCNN. In fact, with better backbone, which is the ResNet 101 variant, the default one uses ResNet 50. With ResNet 101, we get even better results. This DC5 here represents the variant where they add dilation to the last stage of backbone and remove a stride from the first convolution of that stage which gives the name dilated C5 version. Essentially this will increase the backbone returned feature map resolution which is why you will see that these variants will have better performance on smaller objects. Speaking of smaller objects, notice that data ends up having poor performance on small-sized objects, especially compared to FPN variant of faster RCNN in which we use feature maps of higher resolution to improve detection performance on small-sized objects. This is expected because using the coarser feature map deter finds it difficult to predict small objects as well as faster RCNN with FPN. But the interesting thing is that in spite of this slower performance on small-sized objects overall AP of data ends up being comparable to that of faster RCNN. This is because for larger objects data performs much better by utilizing global context and with self attention modeling relationships between objects data does a much better job at predicting large objects accurately. However, if you consider speed data is lacking with YOLO v4. We saw how we were able to do realtime detection. But with data the base model with ResNet 50 is itself not real time. And if we start chasing behind accuracy then with other variants the speed decreases further. With the code that I walk through I get AP50 of around 60 on VOC data set but the authors of DTOR did not experiment on that. However, I did find a paper which is on pre-training in DTR where the authors evaluated the baseline data model with ResNet 50 backbone on VOCC data set. They are able to get AP50 close to 80. I'm guessing this lower performance is because the model that I trained had weaker backbone and that to frozen plus smaller number of queries. In order to further boost the performance, we could add multiscale training which the official repo does and then test on larger sized images. In data repo, the authors test with images of size 800. So that should further improve accuracy after multiscale training. Then we can also make changes in our model to make it stronger with more number of queries, more encoder layers, better and trainable backbone. With all these changes, it should bring us closer to this 80 number. But since that will make the overall training of data much longer, I haven't tried it. However, when you look at predictions made by the current trained model, it isn't doing too bad even with these limitations. Let's now look at some of the ablation experiments that the authors do. First is experimenting with number of encoder layers. And no surprise there as we increase the number of layers in encoder the overall performance of data increases and this result is with fixed number of decoder layers. So we are only changing encoder but obviously with more number of encoder layers the FPS is going to take a hit. Second is regarding positional encoding. For spatial position encoding that is positional embedding for image features. The authors experiment with four variants. Not having them at all. Using fixed sinosoidal position embedding but just at the input, using fixed sinosoidal position embedding but at the attention layers. And learnable position embedding at the attention layers. For decoder, they experiment with learned embeddings at the input or at the attention layers. The best variant ends up to be fixed sinosoidal position embedding for image features at attention layers and learnable embeddings for query objects at attention layers. Next is impact of using NMS and the impact of decoder layers to data performance. As we increase the number of decoder layers, performance increases which is similar to what we saw with encoder. Regarding NMS in data, the authors don't use NMS at all, making the entire detection pipeline much simpler than what we had in our previous videos. But in order to test the impact of NMS, they experiment with and without NMS with data models having different number of decoder layers. When we have small number of decoder layers where the interaction between all query objects is limited, adding NMS surely increases the performance. But as more and more decoder layers are stacked, NMS does not positively impact the detection performance. The fact that DTOR is trained to predict one box per object. This reduces the chances of duplicate boxes which leads to NMS not being that impactful to the final performance. That being said, when I trained on VOCC data set, I noticed that NMS was increasing AP50 performance from 60 to 65. But I think this was because I used the same background weight as the official repo. Even though my data set is quite different that is number of objects per image is different and my number of queries is also different. So I think if I spend some time tuning the background class weight for VOCC data set and with a stronger model I should be able to reach a point where NMS does not add any benefit to detection performance. The last experiment is to understand the benefit of auxiliary loss. DT uses decoder outputs of each layer and computes losses for all of them during training. Though the authors did not mention any results on experiments with auxiliary loss in the paper. In one of the issues on the repo, they do provide results of these experiments on Koko data set using auxiliary losses to train data. It leads to increasing the map from 35 to 40 indicating that using auxiliary loss is very beneficial for better performance of data. Furthermore, comparing the actual value to that obtained with different number of decoder layers, one can infer that without auxilary loss, more decoder layers do not add significant benefit. This was all about results. Now let's move to the last part of the video where we look at some of the visualizations that the authors mention in the paper. When visualizing encoder attention map for the last layer of encoder, we see that the model learns to separate different instances of object even if their spatial position are nearby. The same can also be validated by the encoder attention maps for points on randomly picked images from VOCC data set. This is after training our data implementation on VOCC. The self attention heat map shows that the model learns to separate different instances of objects present in the image. This is true even when the objects are of the same category like this image with person class. and this one with three instances all of car class. The encoder separates the instances regardless of the category being same or different. Same visualization when done on the decoder attention map, we get to see that rather than attending on the entirety of the object, the decoder during cross attention attends only to object boundaries. The authors have the hypothesis that since the encoder has already separated instances, focusing only on the extremities of object is enough for the decoder to localize the object and predict the class. The same is again also found when taking our trained models decoder trained on VOCC data set and visualizing cross attention for the last decoder layer. Focus is always on the extreme ends of the object. Here are some additional examples with more than one ground truth objects. But here also the focus is mostly on the extremities of object like legs and heads of person and horse or wheels of bike. The last visualization that we look at concerns with query objects for Koko data set. After the model is trained, the authors plot the box locations and sizes predicted by different slots. Here they show only 20 out of the 100 slots. Each box prediction is represented by a point with coordinates of the point being same as the center prediction of the box. The color of the point indicates the size of predicted box. Green is small-sized predicted box and red and blue are large horizontal and vertical box predictions respectively. Two things are of interest. First that most slots have instances where they predict boxes that are image wide because that's very common in Koko data set. But the more interesting aspect is that each slot learns to specialize in certain regions. For example, notice how this lot predicts most of the boxes that are on the left side. But this lot specializes more on predicted boxes on the right side of image. And this one specializes in center region. So that means even though we initialize the slot embeddings without any geometrical prior unlike anchors but still through the course of training the model learns to give spatial specialization to these query slots. Same is true for our VOCC trained model. During evaluation on 5,000 VC test images, if I plot the average center coordinate predicted by all 25 query slots, we can notice that the mean center prediction is specially spread out to cover the entire spectrum. X coordinate here represents the average CX value of all the predictions made by that query slot and Ycoordinate represents the average CY value of all the predictions made by that query slot. Some slots specialize in predicting objects on the left side which is why their mean center prediction is on the left. Some specialize in center and some on the right side. same along the other axis. Just to repeat, each point here represents the average box center predicted by each of the 25 query slots when evaluated over entire VOCC test data set. These were all the visualizations that I wanted to present in this video. With this done, we are now at the end of part two of DTOR video. As mentioned at the end of first part because of the length of the entire video I decided to separate explanation and implementation into two different videos. I'm hoping both these videos were worth it and led you to have better understanding of different aspects of deta. Thank you so much for watching these videos and supporting me and have a great day.