Inception Network (GoogleNet) Explained

One major problem in convolutional neural networks is deciding what filter size we should use. Should we use a 3x3 filter or a 5x5 filter or should we use pooling or not? So one idea to deal with this issue is why not try all of them. For instance, on this input feature map of size 32x 32x 128, we could first apply a 1x1 convolution with 32 filters. Then a 3x3 convolution with 64 filters using a stride of one and padding of one so that the spatial dimensions remain 32x 32. Then maybe a 5x5 convolution with 32 filters using a stride of 1 and padding of two so that the spatial dimensions remain 32x 32 as well. This is followed by a pooling layer configured so that the output spatial dimensions also remain 32x 32. And the idea is to concatenate all of these outputs depthwise and feed it forward through the network and the network will learn on its own which of these operations are most useful. This way we wouldn't have to worry about choosing the right filter size or pooling strategy at all. Now there is a problem with this approach and that problem is computational cost. Let's go back and calculate the total number of operations. To calculate the number of operations, we multiply the output height, width, and number of channels by the filter height, width, and number of channels. For the first part, the filter size is 1x 1x 128. And the output is 32x 32x 32. Multiplying these together gives us a total number of operations. Now let's do the same for the 3x3 and 5x5 cases. As for the pooling layer, the number of calculations is so small compared to the convolutions that it is practically negligible. So we will approximate it as zero for simplicity. Adding all of this up gives us a total of around 185 million floatingoint operations. This metric flops is a standard way to represent and compare computational cost in the deep learning literature. So the real question is is there any way to reduce this to something more acceptable? Now if you have noticed the real culprit behind this high computational cost is the depth that is the number of channels. The computational cost is directly proportional to the depth of the feature map or the filter. Now we just calculated the computational cost for the 5x5 convolution alone. It was around 105 million operations. Let's see how we can reduce this using one by one convolutions. So what we do is apply a one by one convolution with 16 filters on the input which reduces the number of channels from 128 down to 16. Note that we are taking 16 just as an example here. There is nothing special about this number. This one by one convolution sitting in between is also called a bottleneck because just like a physical bottleneck, it compresses the information by squeezing down the number of channels. Now if we calculate the computational cost of this 1x1 convolution, it turns out to be just over 2 million. We then pass this compressed feature map through the 5x5 convolution which outputs a 32x 32x 32 sized feature map and the computational cost for this is around 13 million. Adding them together gives us just over about 15 million operations in total which is more than six times less than before. So this is the key idea behind using one by one convolutions to reduce the number of channels and bring the computational cost down significantly. If you are not familiar with one by one convolutions and how they work, I have already made a dedicated video explaining the concept in detail. I highly recommend checking that out first as it will make this part much easier to understand. Now one thing to keep in mind is that we do lose some information due to this channel compression but it is only negligible information and the model does not suffer from it in practice. Now let's build the inception module using everything we have learned so far. For the one by one convolution branch we directly apply the one by one convolution to get the output. No bottleneck is needed here since the 1x1 convolution itself is already acting as an compression operation for the 3x3 branch. We first apply a 1x1 bottleneck convolution to reduce the channel depth and then apply a 3x3 convolution over it to get the output. We do the same for the 5x5 branch. First a 1x1 bottleneck followed by the 5x5 convolution to get the output. All other parameters remain the same as in the naive case. The only addition is the bottleneck before larger convolutions. For the pooling branch, the pooling operation produces an output of the same spatial dimensions and a one by one bottleneck convolution is applied after it to reduce the channel depth. Now one important thing to keep in mind all of these parameters are learned during training via back propagation. So if the model finds that pooling or any particular filter size is not useful, it will automatically learn to suppress it. Finally, all the outputs from each branch are concatenated depthwise and fed into the next layer of the network. And if you now calculate the total number of operations, it comes out to be around 35 million operations, which is significantly less than the naive approach. That covers the core mechanics of the inception network. To dive deeper, I highly recommend checking out the original research paper that the team at Google came up with. In it, they saw both the naive version and the optimized version we have talked about today. The specific model they built using these ideas is called Google Net. It is a 22 layer deep architecture that famously won the Imagenet competition. If you look at the full diagram, it starts with standard convolutional and pooling layers at the beginning. Then it uses a stack of multiple inception bottleneck layers as it goes deeper. You will also notice a few extra branch off layers with their own softmax outputs in the middle of the network. These were added to help the model learn more effectively as it gets deeper. And that's pretty much all for this video.