Designing Network Design Spaces

Key Takeaways

this video will explore a really interesting new paper from researchers at Facebook AI designing network design spaces network design spaces refers to the set of all possible neural networks that can be found in something like a neural architecture search an example of this could be taking an encoding string like 0 1 1 0 2 and then rendering something like a 5x5 convolution with 32 filters and a skip connection to the third layer ahead so before these epically massive evolutionary or reinforced learning searches can find the best possible neural network for a given problem such as Auto regressive language modeling or imagenet classification this design space itself must be constructed to search through so these design spaces are usually manually encoded based on the researchers intuitions about neural network structure this paper instead searches for that design space so rather than producing a single champion Network this produces a design space such that you can just randomly sample quality networks from it this video explored designing Network design spaces from researchers at Facebook AI a really interesting look at neural architecture search with findings and insights generalizable to any kind of meta controller search manually designing neural networks has led to significant gains in computer vision benchmarks like image net classification going from Alec's net to vgg networks to res net and dense net manually introduced heuristics produced better performing networks things like adding more convolutional layers adding the skipped connection putting the skipped connection between every feature map as in the dense net architecture have all shown that finding better ways to design neural networks results in better performing networks in a similar way that neural networks automatically find features describing high dimensional data rather than manually designing the features we're looking to automatically design the neural networks themselves in an area of research known as neural architecture search a core characteristic of neural architecture search is the design space the design space determines all the possible neural networks that the search would be able to produce this paper starts off with the any net design space shown here the design space is biasing the search with the structure of stem body and head and fixed blocks to place in each stage so the search itself is searching through these degrees of freedom that determine things like the depth of a given block the width of the filters layer or the bottleneck feature map compression ratio in a skip connection so this paper isn't focused on producing the single best configuration of these different degrees of freedom that fit into this any net design space rather it's looking to search for the design space itself so searching for different ways of you know having different degrees of freedom that the search is able to choose from when it's you know selecting the parameters that will produce a neural network from this possible configuration space one example where neural architecture search has been really successful is the evolve transformer used in Google's Mena chat BOTS so what they're doing in the evolve transformer is there automatically designing neural networks but the way that they automatically designed the neural networks comes with this manually designed design space so the manually designed the design space the researchers before they begin searching for the evolve transformer they pre encode this or they bias the structure of the neural network to have this left branch right branch you know have the hidden state goes for the normalization layer and then you know have this second layer and then the activation function so what the search does is it takes this manually defined design space and it just searches to plug in different operations different normalizations into the pre design design space so in this paper of the evolve transformer the researchers have manually defined the design space but now we're looking to pop one level up higher in the stack and automatically search for the design space as well as the searching through it that produces the neural network so we're automatically designing this you know predefined kind of parameterization of which the possible neural networks can come out of another interesting paper that's looking at how we're structuring the design spaces from which these neural networks and neural architecture search are driven from is hierarchical neural architecture search this paper is looking at a manually designed design space that makes it so it's quicker to search for these neural networks given things like evolution or reinforce learning search through this hierarchical encoding space but what we're doing in this paper is stepping one level higher of the stack and doing a meta optimization of the design space itself as well as the you know inner loop optimization of finding the best neural network from that design space the authors list some of their goals for designing design spaces simplify the structure of the design space improve the interpret ability of the design space improve or maintain the design size quality and then maintain model diversity in the design space so starting from some initial design space or way of configuring all these different neural networks that we've done just sample a neural network from this design space what they're gonna do is they're gonna iteratively maintain a population of neural networks sampled from a given design space and then try to look into the population manually to inspect it and see if you can you know get any heuristics for how you might want to limit the degrees of freedom in that given design space so the idea is that if you simplify the structure the design space in such a way that the performance isn't damaged by simplifying it too much and then you can interpret it better by getting these insights from a you know a different parameterization like this table described so they're going to iteratively step through the different design spaces sample a population of different networks that are parameterised by the design space that have given iteration step I mean they're gonna imply some restriction on the next step that limits the degrees of freedom makes it so there are less possible neural networks that can be sampled from the parameterization to say going from 10 to the 18 down to 10 to the 8 possible neural networks that can be sampled from the design space the first restriction the authors impose them the any net design space is to make it so they all share the same bottleneck compression ratio in these skip connections so what this does is previously you have this free parameter in the design space that was setting the bottleneck compression ratio differently for each layer so say 3 2 3 1 some kind of distribution of that parameter as you're sampling the neural network so now since they all share the same parameter you no longer have to search for that extra component when you're designing these neural networks so you're reducing the number of total neural networks in the design space by imposing this restriction on the parameterization and in the authors show that this doesn't result in worst performing networks the authors continue to iterate from the original any net initial configuration of all possible neural networks into the Reg net space so what they're doing is in each iteration they're finding some heuristic to impose on the search space that results in better performing networks while also having less possible networks can be driven from the space so for example they find things like when they have these different blocks they each have their own depth sort of in these sub blocks is plugged into the overall network in this kind of you know predefined kind of like macro microstructure they find that doing things like increasing the depth kind of monotonically or increasing the width of the filters monotonically you know monotonically increasing these kind of heuristic results in the better privatization and better producing neural networks after stepping through the any net design spaces they scaled us up to a larger neural network and they find that the same heuristics that produce this reg net space still hold and produce good networks in the you know a higher compute bigger network setting so what they do is they you know iteratively step through these different parameters looking at how they're performing with the sampling of different neural networks from the population to see the performance of different depths different things like the width multiplier or the compression ratio and the bottleneck thing these different kind of parameters that you know define this search space and are the different free parameters that the evolution or reinforcement learning a random search is trying to you know optimize to design this neural network that performs the best on a given task one of the most interesting papers that's come out recently in machine learning research is Auto ml0 in Auto ml0 the design space are these macros set up predict and learn functions these memory addresses that have scalar vector and matrix variables and then these different operations that are manually defined for the evolutionary search to choose how it wants to structure it within this manually defined structure so the way that the designing network design spaces from these researchers at Facebook would do something like this is they would randomly sample a configuration in the design space and then optimize the design space by analyzing the behavior of the population that's just you know when you just randomly sample programs in this case in the autumn l0 case when you're randomly sampling these programs and then seeing how well they perform so it's interesting to think about instead of just optimizing for the single best program optimizing for a parameterization of the search that you can randomly sample from and then still have good performing programs it's interesting to think about the design space hierarchy between this parameterization of neural networks and then the behavior the inner loop optimization of the neural networks themselves another interesting kind of framework to think about is poet where you have the coevolution of the bipedal walking agent the neural network that parameters the controller and then the terrain in which it's walking on itself so it's interesting to think this coevolution teacher-student you know whatever kind of hierarchy you think of this as where you have the design space and then you have the inner loop optimization of the neural network itself so it's definitely interesting to think about you know the coevolution of these different moving pieces and then particularly this meta optimization or you have the optimization of a controller that is optimizing something that itself needs optimization there's also an interesting paper to check out poet with respect to this kind of a you know design space hierarchy thanks for watching this overview of designing Network design spaces a really interesting way to think about the parameterizations of the neural networks and then the ways in which we can look at the neural architecture a search problem I highly recommend checking out this paper to get a closer look at the details and the findings that they have from stepping through these different design spaces and then you know plotting the different errors of random sampling populations from the different design spaces and then making these heuristic restrictions on the neural networks that can be driven from the next step of design space iteration thanks for watching and please subscribe to Henry AI labs for more deep learning and AI videos [Music]