ResNet Explained!

Key Takeaways

[Music] this video will explain the resident a deep convolutional neural network architecture design this is one of the most popular neural network designs that have ever been published with over 20,000 citations deep learning is thought of as learning a hierarchical set of representations such that it learns low mid and high level features in images this is analogous to learning like edges and then shapes and then objects so theoretically more layers should enrich the levels of the features and previous models to the ResNet typically have depths of 16 and 30 layers so the idea is that shouldn't building better neural networks because easy as adding more layers to the network so the first contribution of the ResNet paper is showing that if you just continue to concatenate convolutional layers on top of activations and batch normalization the training will eventually get worse not better but they offer this insight the construction insight that says if you consider a shallow architecture and it's deeper counterpart with more layers theoretically all the deeper model would need to do is to just copy the output from the shower model model with identity mappings so the construction solution suggests that a deeper model should produce no higher error than the shallow counterpart however the identity functions aren't an easy function to learn and so therefore the residual functions formulate the layers is having a reference to the input through these identity or skip connections such that theoretically if it needed to push the layer down to zero it could easily do it in this framework so again this shows the residual connection which is the building block for the residual network or resident so one interesting thing with res Nets is if the previous layer dimensions don't match the input to the next layer you think about a convolution a 3x3 convolution would change the spatial dimensions of an image from like 32 by 32 to 30 by 30 so they do here is they propose different schemes for up sampling the previous input layers to through this identity skip connection so one of the two ways it can do this is it can either just zero pad the outsides of the spatial dimension and there's no extra parameters with doing this it's really quick or I can expand the dimensions with one by one convolution so this image shows what the resident looks like in contrast to a 34 layer plane Network which is a series of convolutional layers followed by activations followed by batch normalization and compared to another really popular model the vgg 19 so the resident experiments they test the 152 layer net on image net and this gets their state-of-the-art results and this is eight times deeper than vgg nets but in terms of the floating-point operation measurement it actually has less less computation and the vgg 19 shown by these billion flops metric so the ensemble event resonance they're able to achieve three point five seven percent error on the image net test set which achieves them the state VR they also test this on C 410 with a hundred and a thousand layers and then they use the ResNet features on the cocoa object detection so with the object detection networks work is you would use like VG or ResNet to extract the features from the image data set and then you would classify the different bounding boxes based on some region proposal algorithm so some more details about the resident is that it uses a batch formalization after each convolution and before activations it uses the hey initializer invented by the author of the paper climbing hey you use the batch size of 256 they have this learning rate scheme the weight decay and then also interestingly they don't use dropout and they have they test on this one interesting test time augmentation where they don't just predict on the test image what they do is they take ten crops from the test image and then they predict the model predicts on each of the crops and then they average the prediction that to form the final prediction so the first experimental result is showing how the ResNet continues to get better as you go from 18 to 34 layers but the naive concatenation of convolutional layers is already starting to get worse so just say 27.8% to 25 percent error rate whereas the play network goes up almost half a percentage from the increase in layers so then they test this idea of when you're skipping ahead and the dimensions don't match to you zero pad it do you have these one-by-one convolutions and how frequently do you use the world above all collisions so they do find that when they have 1x1 convolutions or known as projections that they do get a slightly significant performance boost but it comes with the cost of having a significant amount of extra parameters so one of the thing they do is when they train the present at 5101 and once these two is they extend the Skip connection so it skips ahead two layers rather than one like the normal residual building block and this is done just to save training time so these are the results of the different levels of ResNet the B and C denoting the different ways of doing the projection matching and then compared to some of the other said via our models like Inception vgg yeah so this is the results of the ensemble of resonances at the state-of-the-art on the top 5 predictions on the image net test set and also these are the results from the CFR 10 data set and interestingly in this is you see that when they try to go from 110 layers they achieved the city art with this but when they try to go to 1202 layers the error goes back up so they haven't quite figured out how to make it go that deep yet and then this shows how using the features extracted from ResNet outperform vgg on the localization or the bounding box detection task so again they find that when they try to they do figure out how to make it significantly deeper than like the vgg 19 layers but with this mechanism the 1202 layer and that still does not perform well and they suggest in the paper that this is due to overfitting so thanks for watching this video on rez nets please subscribe to Henry AI labs and the paper link is in the description [Music]