RevNet: Backpropagation without Storing Activations

Key Takeaways

[Music] this video will explain the reversible ResNet architecture the red net the key idea here is to do back propagation without storing the intermediate activations so the headline is that the red net is an algorithm to restructure a convolutional blocks such that the activation doesn't need to be stored so typically as you would do a forward pass through the network you would store all those intermediate activations to use this back propagation so with the red net restructuring the convolutional block you're trading off computation for more memory so the here's the idea of the reversible block you split the input X into two inputs X 1 and X 2 along the channel dimension and then y1 and y2 are also divided in this way and they're constructed from the each of the x1 next to so this diagram shows what the forward pass this would look like the x1 and x2 are split along the channel dimension and then they're used to compute the y1 and y2 such that if you have Y you can just split along the channel access and then use this to derive X 1 X 2 and thus X so in this sense if you have Y you split it up into y 1 and y 2 and then you use these intermediate functions to get back the X 1 X 2 so this way you don't need to store the intermediate activation so you just need to store the activation from the very last layer and then you can use this propagation rule to derive the activations throughout the network so overall this is the red net back propagation algorithm and so the key difference here is that you have to do like another forward pass so usually if you needed any computation for the forward pass and to end for the back prop now you're gonna need 3n for the back prop because you're gonna have to do one more forward like computation in each of these blocks to derive the input from the activations so this is overall this is 33% more computation than the original block would be so here's a comparison with other there there are some other techniques that they mentioned I didn't really get into the details of them but this is like a state of the art in the spatial complexity of storing activations you only need a constant amount of memory for the activations in the very last layer of the network so these are some of the architectures that they tested they have the ResNet 32 compared to this modification RedNet 38 and then did you know other slight modifications and then they test on the CFR 10 and 100 and imagenet data sets so this is the big impressive performance that they present is that it's only slightly worse than the ResNet on CFR 10 CFR 108 even when you restructure it in this way and the reason that it's slightly worse and not you know exactly the same is because there's some numerical error when you're approximating it in this way but I didn't really completely get into the details of that so this is the ResNet block and so in this case it's f that we're replacing with this reversible block you can also imagine if you have like a channel wise concatenation like in the resident box usually it's the next features are X plus f of X so like you're adding it at each location you're adding the features we could also imagine like concatenating and kind of just like stacking along the previous layers features along the channel dimension so if you did that it's really straightforward to see how you could reverse it as well because if you have this layer Y these activations Y and then they're split such that it's y1 and y2 and y2 is two previous layers activations stacked onto the back of this y1 then you can easily just take the y2 and then you can pass it through the F function to get out to y1 so this is just sort of like an introduction to how this reversibility can work to allow you to go back into the network in different ways of structuring your networks so the big takeaway is that is 33% more computation required but you save yourself a ton of memory so this is really useful for a larger batch sizes networks and just the size of your intermediate feature map stick and what not a frequent problem that I encounter is running out of GPU memory and this is a really great method for avoiding that so thanks for watching this video from Henry AI labs please subscribe for more videos on deep learning