[Classic] Deep Residual Learning for Image Recognition (Paper Explained)

#ai #research #resnet ResNets are one of the cornerstones of modern Computer Vision. Before their invention, people were not able to scale deep neural networks beyond 20 or so layers, but with this paper's invention of residual connections, all of a sudden networks could be arbitrarily deep. This led to a big spike in the performance of convolutional neural networks and rapid adoption in the community. To this day, ResNets are the backbone of most vision models and residual connections appear all throughout deep learning. OUTLINE: 0:00 - Intro & Overview 1:45 - The Problem with Depth 3:15 - VGG-Style Networks 6:00 - Overfitting is Not the Problem 7:25 - Motivation for Residual Connections 10:25 - Residual Blocks 12:10 - From VGG to ResNet 18:50 - Experimental Results 23:30 - Bottleneck Blocks 24:40 - Deeper ResNets 28:15 - More Results 29:50 - Conclusion & Comments Paper: https://arxiv.org/abs/1512.03385 Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers---8x deeper than VGG nets but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. D