Equivariant Neural Networks | Part 2/3 - Generalized CNNs

Key Takeaways

[Music] welcome back to the second part of this equivarian deep learning Series in this video we want to have a closer look at some of the model architectures that integrate equivariants with respect to specific groups of course it's impossible to give you a complete overview of the literature because there's just too much out there but I've picked out some interesting papers that will certainly be a good foundation also note that if you read some of the papers it's quite likely that the math around group Theory and representation theory is confusing at first but I hope this video and some other resources I mentioned at the ends will help you to master it I decided to extend this series with a part 3 and dedicate this video to generalizations of CNN's only equipped with some basic knowledge about group Theory from last video let's begin with General Group convolutions by Taco Cohen and Max Belling start on the same page recall that classical CNN simply map pixels to feature Maps which are stacked based on a number of filters that are used these models perform template matching meaning that they use the small learnable filters to search for similar patterns on the image as we found out before cnns are translation equivariant default translations of the inputs lead to translations on the feature Maps the convolution operation can also be expressed mathematically as follows it's defined as the inner product of a signal such as an image and a kernel k usually we have multiple kernels and therefore we have an additional sum in this expression for the number of channels what all of this means is that our input signal is evaluated at some position Y and multiplied with the kernel which is translated by different amounts in the pixel space this expresses the popular sliding of the kernel over the image in Signal processing you will find a slight distinction between convolution and cross-correlation convolutions additionally flip the filter before applying the signal this is also nicely visualized in a recent three blue one Brown video Linked In the video description in our deep learning setup it doesn't make a difference however as the filters are anyways learned by the model group convolutions now simply replace the shift with a more General transformation from some group G as a result each pixel is moved to some new position which is determined by multiplying with a group transformation for instance the G is some rotation Matrix which is Multiplied with each position y in the pixel space you might wonder why it's called G to the power of -1 typically we Define these group Transformations with respect to images but in this definition we operate on pixels therefore we keep the image fixed and instead shift the pixels Fabian fox has a great visual of that in his blog post on equvariance and this notation simply means that moving the pixels in the opposite direction will move the whole image in the desired Direction so overall it's just a notation thing and the key takeaway is that the kernels are not only translated like in classical convolution but instead transforms according to other group symmetries now what is the output of these group convolutions a nice way to think of group convolutions is to say that they are data augmentation on the filters this is also because it's more efficient to transform small filters instead of the whole input this example from the paper shows the output of group convolutions for the 2D 90 degree rotation group together with Reflections which is called p4m so for each possible group elements we get an activation outputs this is basically the feature map of a group convolution it resembles all group elements and often this is therefore called a structured feature map you might also see the name filterbank and now the part where it gets abstract after one of these layers our base space is now not anymore the pixel space but instead a group space so this structured feature map if we want to perform group convolution on this group again we need to perform another set of group actions on these group representations as a result the notation for subsequent layers slightly changes this is the definition for group convolutions after the first layer you can see that we operate on group representations H now instead of pixel as before what this means is simply that some already transformed output is transformed Again by another group action G the transformation of such a group representation is done by individually transforming each of the kernels and transforming the whole structured feature map this is best visualized in the following blog post I've Linked In the video description for example here we Define convolutions for the 2D 90 degree rotation group so we have four possible rotation elements we start with some input image which is duplicated four times for better visualization in the first layer the images on the left are multiplied with the kernels where each of them is transformed according to the rotation group in the second layer each kernel is rotated individually and additionally all of the kernels are rotated for each of the four positions if you want to try out group convolutions in practice there is a python Library by Taco Cohen called crew pie it also includes a pytorch implementation and here's an example of how the group convolution layers can be set up so you just have group specific convolution classes that can be used like every other convolution module as far as I understood it the implementation simply uses the classical conf2d module of pytorch and applies it to all of the group elements now so far we only had a look at rotation Equity variants and didn't explicitly talk about combining it with translation Equity variants of course this is happening under the hood because we typically combine different group transformations in group Theory you call this combination of groups a semi-direct products there are many examples of using multiple groups for example this paper for medical image analysis the kind of group convolution they use is called se2 the special euclidean group so translation and rotation in 2D in the paper they call these layers lifting layers as they lift up a 2D image to a 3D filter bank they show that this kind of Architecture is beneficial for many Medical Imaging data sets where the object of interest can be rotated in various ways of course using equivalary models is not only more data efficient but also parameter efficient because of the increased weight sharing here you can see what is happening under the hood in this paper first the 2D input is translated like in classical cnns and rotate it according to some discrete rotations of a rotation group this leads to a so-called se2 image which is this 3D filter Bank as you can see the response for each rotation and translation is stored and this generates an additional dimension for each rotation group elements then they project it back to 2D by applying a maximum operation at the end over the rotation Dimension Theta so far so good in this example the size of the filter Bank is still acceptable but as you can imagine for larger groups or even infinite groups like continuous rotation this approach is not feasible this brings us to a further generalization called steerable cnns published by Cohen and dwelling they present a theoretical framework that enables the implementation of group convolutions without having to discretize groups as a warning when you dive into this area and have only few knowledge in mathematical representation in group Theory you will probably not understand a lot in the beginning that was at least my experience foreign [Music] the fundamental idea of these models is to use steerable filters this is an idea that was already developed in computer vision back in 1991 and here is simply integrated into cnns these filters can be constructed as a linear combination of Base filters the advantage of this is that we don't need to store the result for every group elements and instead can construct them by combining the output of these base filters this decouples the size of the group from the computational costs this is the overall intuition but first we need to understand a couple of theoretical aspects first of all instead of talking about stacks of transformed features we need to think of fibers so usually the outputs of one filter are thought of as a channel but here the output of a filter at each pixel location corresponds to a vector such a fiber is an independently transforming unit often these fibers are referred to as feature Vector Fields as we assign a vector to each position X now how does this help us such Vector Fields also encode the notion of orientation and when applying group Transformations on these fibers we not only move them to a new position but also change their orientation the idea now is to use steerable kernels to decompose these fiber representations such that we have control over how they move in the Target space in other words we can steer the output Vector representations and how they change under transformations the formal definition of searability is a bit theoretical and that's why I left it out for this video but the idea is just that we find a linear transformation of the output fibers that lets us control how the fibers transform when the whole image transforms the last important component of steerable cnns are so-called irreducible representations or short errifts these are the basis filters that represent Elementary building blocks of the output representations in the paper they show a discrete example for the Roto reflection group D4 for this specific group it is now possible to express any equivalent filter Bank as a linear combination of these spaces representations the big Advantage here is that the number of parameters only depends on these spaces elements and not the size of the group anymore to be honest with you I didn't go further into details here as there were a couple of things that I didn't fully grasp but if you want me to do a separate video on this paper please let me know in the comments foreign [Music] filters can also be found in follow-up work for example in harmonic Networks here Equity variants 2 continuous 360 degree rotations are modeled using a linear combination of Base filters these base filters belong to the family of circular harmonics the shape of these filters is determined by a real and an imaginary part which are learned during training these space filters are also called complex gaussian filters and can be expressed with polar coordinates the final piece here is that the output of an unrotated image is the same one as the output of a rotated image [Music] one last Model I want to mention in this video are capsule networks or short caps Nets because they are somewhat related to steerable cnns they were originally presented by Hinton in 2011 and this paper from 2017 presented a technique called Dynamic routing that allows to combine the outputs of different capsules so what is a capsule and why should we care about them capsules are motivated by the brain which is organized into modules a capsule is a group of neurons that take a vector as input and output a vector as well this is already different from regular cnns which output a scalar value capsnets therefore encapsulate information into small vectors and because they do this they are equivariant to changes of the inputs as they individually maintain position orientation and post information true mathematical equivariants is however not guaranteed as follow-up research showed the output vectors of capsules are then combined with the aforementioned Dynamic routing this is basically what the max pooling in CNN does in a popular talk by Hinton he said that the max pooling in cnns is a big mistake and caps Nets were the proposed solution to this routing coefficients between the capsules are learned by the model and instead of using the max value the model is able to more intelligently combine information for the next layer so you can say that the dynamic routing is a generalization of this Max pooling aggregation this routing allows to combine information in a hierarchical way and higher capsules in the later layers pick up on information provided by the lower capsules so just like in steerable cnns we also have a vector Fields here that consists of independent disentangled units that are invariant to certain transformations in fact in steerable cnns the output fibers can be seen as a stack of capsule outputs with the additional property of durability of course all of the things I just mentioned in 2D also exist in 3D for example there are also 3D steerable cnns that operate on 3D Fields as well as spherical cnns that Define convolutions on spheres there's also a nice YouTube video on 3D steerable cnns that might help to understand what's happening there are many helpful videos and blog posts I used to build this video series and if you look for more in-depth explanations I can highly recommend these three resources first of all the introduction to group equivariant deep learning by the University of Amsterdam a part of this course is also a very detailed YouTube series on a variety of different models Fabian folks has several nice articles about equivariants that I also highly recommend finally there is a book called steep learning for molecules and materials that includes two chapters on Group activariants which I found very helpful this video was all about CNN generalizations and I really have to say that I was surprised how many models exist that totally make sense but are not that commonly used so overall I hope this high level overview was useful for you and I'll look forward to seeing you in the last and final video about equivariant deep learning [Music]