Self-Tuning Networks: Amortizing the Hypergradient Computation for Hyperparameter Optimization

Key Takeaways

all right hi everyone and welcome to the autumnal speaker series i'm really really excited to introduce roger grosse who is an assistant professor of computer science at the university of toronto and a founding member of the vector institute roger received his phd in computer science from mit and then spent two years as a postdoc at the university of toronto he holds a canada research chair in probabilistic inference and deep learning an ontario mris early researcher award and a c for canadian ai search chair i'm in general a huge fan of roger's walk but i'm particularly excited by his work on self tuning network which is going to present today lucky lucky coincidence uh as a reminder roger will be around during the talk to us to answer questions uh during the talk i will also be clustering and aggregating questions to take to our live q a at the end so without any further ado let's see roger's talk i'm roger gross and i'll be talking about self-tuning networks a way of amortizing the hyper gradient computation for hyperparameter optimization this talk will be based on two published papers the first one is self-tuning networks by level optimization of hyper parameters using structured best response functions this was primarily the work of three students matt mackay paul v cole and john lorraine also in collaboration with david duvenau and the second paper is delta scn efficient by level optimization of neural networks using structured response jacobians this is primarily the work of my student jooheon bay and this is published at nurip's 2020. so i'll begin by observing that a lot of problems that we're interested in solving in machine learning especially hyper parameter optimization have a bi-level structure in the sense that there is an optimization objective the outer objective which is itself defined in terms of the optimal solution to an inner objective so in the context of hyperparameter optimization the inner optimization problem the one shown on the right would be to optimize the weights of a neural network to minimize the regularized training error the outer objective shown on the left would be to find the hyper parameters which minimize the validation error of the optimal weights as far as the inner objective so this is an example of a by level optimization problem and there are various techniques for solving them which i'll summarize later but a lot of them require the gradient of the outer objective this is what i'll refer to in this talk as the hyper gradient this is what we need if we want to do any sort of gradient based optimization of the outer objective and computing this hyper gradient is a big computational challenge required for many different types of machine learning algorithms even outside by level optimization and so computing this hyper gradient will be the focus of this talk so one way to approach bi-level optimization problems is not to use gradients at all and just treat them as a black box optimization problem so if you're tuning hyper parameters probably the vanilla ways of doing that would be with grid search or random search where you just try a bunch of values and train networks for each one and pick the one that does the best on the validation set these are non-adaptive approaches so they don't make use of information that you've learned from previous queries a somewhat smarter way of doing the black box optimization is using an adaptive method such as bayesian optimization so this actually tries to use information learned from previous queries in order to choose points that are both likely to perform well but also provide useful information about the cost function so evasion optimization is great it's been a key enabler of many applications such as alphago but it has some drawbacks so one drawback is that like other black box optimizers it doesn't exploit gradient information at least not the commonly used versions of the algorithm and the other drawback is that it's an offline procedure which means it requires training the algorithm many many times typically from scratch so we can often do better if we make use of gradient based optimization and in order to do that we need to compute the hyper gradient and to actually create many different applications and machine learning that somehow make use of the hyper gradient so the canonical one that i'll be using as the running example throughout the talk is gradient-based hyperparameter optimization but there are many other cases that we might not even consider by level optimization such as influence functions used to figure out which training examples are useful or relevant to a particular prediction and neural architecture is based on implicit layers and equilibrium models and cases like that so there are many different cases where we would like to have a hyper gradient and so hyper gradients as i'll describe in a moment are impractical to compute exactly and so in almost any paper that you see that uses hyper gradients they're almost always computed using one of two methods the first one is implicit differentiation so you can apply the implicit function theorem to get an expression for the hyper gradient but this expression involves a very large linear system you have to solve a linear system involving the hessian of a neural net or some matrix that's closely related to it and this is typically done using an iterative procedure something like conjugate gradient neumann iterations or the lisa algorithm so that's one way you can do it another way is to unroll the inner optimization so that gives you a differentiable procedure that you can back prop through in order to get the derivative with respect to the hyper parameters and so these are the two methods that are almost always used to solve the problem and they both have the disadvantage that they both require expensive iterative computations each time you want to update the outer variables and so the goal of this talk will be to present a third alternative that you can consider for hyper gradient computation and the way that this alternative will differ from the other two is that we amortize the hyper gradient computation by learning a hyper network so you invest some computation into learning the hyper network and then the individual gradient computations for the outer objective should be fairly cheap and this will allow you to adapt hyper parameters online as the neural network is training and so the procedure i'll define will turn out to have only a small constant factor overhead compared with training a single network so it will require about twice the amount of memory to store the parameters as the elementary neural network and require a small constant factor the amount of compute time to train the network and so i've described a high level umbrella picture of the different places that hyper gradients are used but now in order to make things a bit more tractable i'm going to limit the scope so the first way i'll limit the scope is that we are dealing with neural networks so i'll talk about a way of approaching the hyper-grading computation that exploits the structure of neural networks a second limitation is that i'll be focusing on outer optimization variables or hyperparameters that appear as part of the inner objective so this would include things like regularization hyperparameters such as dropout or data augmentation rates but it rules out dynamical hyper parameters which aren't part of the training objective but influence maybe the efficiency of training so things like learning rates and momentum hyperparameters and things like that so we're not considering those for now and this also has the implication that for hyper parameters like dropout they've been shown to have both explicit and implicit regularization effects they have imp they have explicit regularization effects insofar as they change the cost function that is being optimized and they also have implicit regularization effects insofar as they change the amount of gradient noise and so in this talk i'll only be considering the first effect and not the second and so we might find sub-optimal hyperparameters to the extent that we're not accounting for implicit regularization and accounting for the implicit regularization is still a very hard problem i'll narrow it down in another way which is that we consider only a single inner objective that's being optimized so we're trying to train one neural net using one training loss so this rules out certain types of by-level optimizations such as implicit layers of neural nets where you have a different inner objective for each data point that you visit and finally we're going to limit the scope to relatively small numbers of outer variables so optimizing perhaps dozens of hyper parameters rather than millions of hyperparameters as some other papers have attempted to do so these are the limitations that i'm putting on the method and broadly categorize approaches to biolevel optimization on two dimensions so i've already distinguished black box versus gradient based optimization do we use the gradient information or not and we can also distinguish offline and online optimization right do we use an offline procedure that requires many runs of the inner optimization in order to solve the problem or do we want online procedure that can be done online in one training run so that you can solve the by-level problem in a comparable amount of time to just training a single network and so our approach the self-tuning networks will be in the bottom right quadrant it's a gradient based approach to online hyperparameter adaptation so i'd like to contrast the bi-level optimization setting with another setting that's commonly used in deep learning which is simultaneous optimization this is something that's been very successfully used for things like gans and neural architecture search here you have two different neural networks or two different sets of parameters that are being simultaneously optimized to two different cost functions and in this setting the aim is to find a nash equilibrium where each of the two players thinks that its solution is optimal for its own cost function without thinking about how the other player will react to it but i claim that this is the wrong solution concept for hyper-parameter optimization and the reason is that it only takes into account when you're optimizing the outer parameters it only takes into account their direct effect on the outer objective the validation loss so in the case of hyper parameter optimization if you consider a hyperparameter like a dropout rate this doesn't appear directly in the validation objective because dropout isn't used at test time and so the direct gradient of the validation loss with respect to the dropout rate will actually be zero and so we have no way to optimize that hyperparameter with simultaneous optimization and even in the setting where the gradient is non-zero this simultaneous optimization setting will ignore some effects that we might care about so for instance if you're doing neural architecture search and the outer optimization variables somehow parameterize the network architecture the simultaneous setting might be able to make a trade-off between the performance of the network assuming the current weights versus the computation time but it won't consider the regularization effect of having a more constrained architecture right having fewer units per layer might encourage the network to learn a more efficient representation so i claim that what we actually want for settings like hyper parameter optimization is a sackleberg equilibrium so this is the game solution where the outer player goes first and considers how the inner player will react to its choice when we're tuning the hyper parameters we want to think about how the weights of the network will react to the hyper parameter choices and so a key concept in thinking about bi-level optimization is the best response function this is the function mapping from the outer variables i'll be using the hyperparameters as the running example so so it's the function mapping from the outer variables to the optimal inner variables for the inner objective and so i will write that as the function r of lambda which is just the argument of the training loss with respect to w and so if we somehow had access to the best response function then we could do gradient based optimization over the hyper parameters because we can simply write the validation loss in terms of the best response function and so what we're showing on the right is an example of the inner objective defined as a function of both w and lambda and the best response function is shown with the solid green line okay so if we want to do gradient based optimization of the outer objective we're going to need to compute the total gradient of the validation loss with respect to our hyper parameters and so when we apply the chain rule we get a sum of two different terms the first term is the direct gradient this says how perturbing the hyperparameters will directly affect the validation loss right you write down the formula for the validation loss and compute the partial derivatives of that with respect to the hyper parameters so this is the term that's being considered in the simultaneous optimization setting the second term is the response gradient so this determines how changing the hyper parameters changes the validation loss insofar as it changes the optimal solution to the inner objective so this is the thing that we really care about when we're optimizing the hyperparameters and so if we look at the formula for this this can be rewritten using the chain rule as the response jacobian times the derivative of the validation loss with respect to the weights the response jacobian says how the optimal weights change if you perturb the hyperparameters so we have a formula for the response jacobian at the bottom of this slide and unfortunately this formula involves the inverse of the hessian for the neural network the hessian of the training objective and unfortunately this makes it impractical to compute because the hessian is a gigantic matrix the dimension is the number of parameters of the neural network which could easily be in the millions or billions or perhaps even trillions so approximating this linear system is a significant challenge visually we can understand the response jacobian by taking the first order taylor approximation to the best response function so that's shown as the dashed green line okay so how do we actually approximate this hyper gradient so i'll be talking about the main technical contribution for this talk which is self-tuning networks that's the idea behind this approach is that at any given time we're going to have a guess lambda of the optimal hyper parameters and what we'd like to do is train the hyper network to approximate the best response function accurately in the vicinity of these hyperparameters and if we have that then we can cheaply approximate the response jacobian just by differentiating through our hyper network that can be done with ordinary backprop this is exactly what we need to compute the hyper gradient and if we have already gone through the work of learning this hyper network then it actually makes the hyper gradient computation very cheap instead of doing an expensive iterative procedure to solve a large linear system we can just do backprop through an ordinary neural network all right but um how do we actually learn this hyper network and so the basic idea which was already pioneered by a paper by john lorraine and david duvano in 2018 is basically to train your hyper network like an ordinary neural network so for each batch of data you sample a new vector of hyper parameters and the hyper network predicts a weight vector perhaps conditioned on those hyper parameters and you do a gradient update on the hyper network in order to reduce the regularized training loss on that batch and so it's basically computing the things that an ordinary neural net would except that it can perhaps perturb the weights of the network in order to account for the changes to the hyper parameters so conceptually this sounds pretty good but this leaves the big question of how do we compactly approximate this best response function right this is the mapping from hyper parameters to the weights of the elementary network this is a really complicated high dimensional nonlinear mapping and in fact the output dimension of this mapping is d the number of weights of the elementary network so you can imagine it being really hard to represent such a function so there's a partial solution we can restrict the hyper network to be linear we have a formula for the linear hyper network that's just a linear function of lambda why can we get away with this well the idea is that the only thing we need the hyper network for is to compute the response to jacobian and the response jacobian is a linearization to begin with and so perhaps a linear architecture is sufficient to compute it and so this argument is sort of true it has some subtleties that i'll get into later on in the talk but i think linear hyper networks seem to be fairly effective but unfortunately this doesn't solve the problem because the dimensionality of the hyper network is still huge to represent a general linear function from hyper parameters to the weights of a network this will require d times h parameters where h is the number of hyper parameters so this is still too large to practically represent so i'll start with another observation which is that best response functions can actually have a very particular structure that we can exploit and so consider for instance the problem of linear regression with an l2 regularizer so here we have the formula for a linear regression objective and we're penalizing the l2 norm of the jacobian of the prediction function equivalent to the l2 norm of the regression weights and that penalty is scaled by the x of the hyperparameter lambda and it's actually possible to derive an optimal solution we can derive an explicit form for the best response function in terms of lambda and so the optimal weights can be written as a product of a matrix q times the vector s and the matrix q can be written as a fixed matrix which is multiple each row of this matrix is multiplied by a value that depends on the hyper parameters so it's a fairly simple equation that exactly captures the best response function globally and the interesting thing about this is that the only effect of the hyper parameters is to rescale the individual features in this case the features come from the principal components so if we interpret that solution as the neural network it looks something like this so we have a linear neural network takes in an input x and produces a scalar output y and it consists of two linear layers the first does a matrix vector multiplication by the matrix q zero and the second layer does a vector vector multiplication to produce the output y and the activations in the first layer are modulated based on the hyper parameters lambda so we have modulatory connections which multiply each of the activations by values that depend on lambda and so the intuition here is that the hyper network reacts to the hyperparameter by emphasizing or de-emphasizing different features so if it sees a large value of the regularization parameter it has the capability to de-emphasize a certain feature so i think this is a generally useful architecture for representing hyper networks and so our self-tuning network architecture combines both of the aforementioned architectural insights so we make the hyper network linear in lambda the weights are linear as a function of hyperparameters now this isn't a linear network it's not linear in the inputs it's linear in the hyperparameters so we combine that in sight with that of modulatory connections and so each weight layer implements the simple functional form which is closely inspired by the optimal solution for linear regression so we have the ordinary neural network computations except that the weights are the the incoming weights to each unit is are rescaled by a value that depends on the hyper parameters and so if you count the parameters and computations the number of parameters is little more than twice that of the elementary network because we have two copies of the weight matrices instead of one and the computational cost is slightly more than the elementary network we have to do a little bit of work to manipulate matrices that are the size of the parameter matrices but the actual feed-forward computations done by the network so if it's a convolution network the number of convolutions and things like that that's the same as ordinary training so the amount of computational overhead is actually fairly minimal okay so how do we train the hyper network so let's first imagine that the hyperparameters lambda are fixed rather the distribution of hyperparameters is fixed and we want to train the hyper network so the hyper network's training objective is just the regularized training loss so it's the expectation of the regularized training loss with respect to the hyperparameter distribution so for each batch it receives a hyperparameter vector it outputs the weights of the network and then it's evaluated on the regularized training loss with those weights and so we simply do ordinary gradient descent and ordinary backdrop to update the hyper network okay so that's the hyper network and we've assumed so far that the hyper parameters lambda are fixed but how do we actually adapt lambda and remember that we're sampling the hyper parameters from a distribution for each batch of data and so assuming a gaussian distribution we also have to choose the scale of that distribution right how large a perturbation do we apply to the hyper parameters for each batch and so when we adapt the hyper parameters we have two different goals there's the ultimate goal of this training which is that we would like a hyperparameter vector that leads to good validation loss at the end of the day but we also have to worry about how this distribution affects the training of the hyper network so we would like to choose values of lambda and sigma that give a useful distribution for updating our hyper network and so what we do is we adapt lambda and sigma simultaneously to updating the hyper network we adapt lambda to minimize the expected validation loss and we essentially do that by differentiating through the hyper network and so we're implicitly making use of the response jacobian the so choosing the perturbation scale sigma is a little bit more subtle we want to choose a perturbation scale that is broad enough to give information about the response jacobian but not so broad that it becomes impossible to represent the hyper network with a linear function so in the left-hand figure we have an example where the distribution is too narrow and the hyper the hyper network can minimize its objective without capturing the jacobian in the right hand figure we have an example where the perturbation scale is too large and so the hyper network has to fit a region that is too large to be modeled with a linear function but if we choose the intermediate value of the perturbation scale like on the middle figure then we can get a good estimate of the response jacobian and so how do we adapt the scale if we simply adapted it to minimize the validation loss like we did for lambda then the optimal value would be zero because there's no advantage in terms of validation loss to making things stochastic and so in order to encourage a broad perturbation distribution we regularize the entropy of the distribution we encourage the distribution to have high entropy which encourages large perturbations but at a certain point the perturbations will be pushed to be smaller because they'll start to interfere too much with the validation loss and so the algorithm essentially trains all three of these sets of parameters jointly the hyper network the hyperparameter lambda and the perturbation scale sigma so everything is essentially trained as a simultaneous game and so how does this work so here we have an example of adapting the dropout parameter for a rnn language model on pen tree bank and we see over the course of training that the dropout rates tend to start out small and get larger and this this isn't just an effect of the adaptation uh taking a long time to converge because the left hand figure shows the whole process starting from five radically different initializations going all the way from point zero five up to point nine and we see that regardless of the starting value the initialization is unlearned very quickly and all of these training runs quickly follow the same schedule and what we find is that this schedule actually works better at the end of the day than any fixed hyper parameter setting so if we compare to a grid search of the same architecture trained with individual dropout rates the stn can beat all of them and it's not just an effect of the stn dynamics this is actually apparently a good dropout schedule because using the crop schedule to train a network outside the context of stns also beats all of the individual settings here we have some examples of training convolutional networks for image classification on cfr10 and we're adapting 15 hyper parameters that include layer specific dropout rates including all layers of the network plus the input layer obtaining the amount of input noise various kinds of data augmentation and we see that the hyper parameters adapt in very different ways it tends to choose large dropout rate for the very top layer of the network a standard trick in training confnets it turns off dropout and a lot of other layers completely and at the bottom of the slide we're comparing the validation loss achieved in a certain amount of time for stn versus other kinds of hyper parameter optimization grid search random search and bayesian optimization and the fact that everything is optimized in a single training run means that the scn is able to find a competitive hyper parameter setting much faster than any of these other methods okay so that was the original self-tuning texture and now we want to tell you about some improvements that we've made to it and this is the work of jooheon bay presented at nurbs 2020. so our delta stn is the result of some algorithmic improvements that we made to the basic scn approach so the two improvements are a centered parameterization of the hyper network and a linearization of the network computations and i'll describe in a bit what problems these are solving and why they help and so we find that the delta sdn significantly improves the speed and stability compared with the original stn and in this presentation i'm going to describe the improvements using a full linear hyper network just to keep the notation simple but in practice we actually apply these techniques to a compact architecture based on the original scn so still low overhead compared with training the original network okay so the first trick is to use a centered parameterization of the hyper network so on the left hand side we have a full linear hyper network which i use to motivate the scn architecture on the right hand side we have the centered parameterization of the same network and these are essentially they're equivalent in terms of the representational power they can all represent general linear functions but the difference is that the hyperparameters are centered by subtracting out the mean value lambda 0. and the centering results in a nice intuitive separation between the two different parts of the hyper network we have the intercept term w0 which you can think of as the current weight of the network right these are what the algorithms thinks are the optimal weights for the current set of hyperparameters the other set of hyperparameters is the linear part of the hyper network these parameterize the response jacobian how the weights should change in response to a perturbation to the hyperparameters all right and so why are centered parameterizations useful well there's one reason that a lot of you might be used to already from training neural networks which is that centering things often improves the conditioning of a cost function so there is a classic analysis by john le in 1998 which you can also find in the first chapter of my neural net training dynamics course notes you can basically understand how centering inputs or centering activations affects the conditioning of the hessian of a cost function and so that's the kind of standard thing in neural net optimization and you know often makes maybe a subtle difference in the training but it turns out to be really important for learning these hyper networks and the reason is that the uncentered representations create certain harmful game dynamics in the optimization and so why does that happen well if we think about the original scn parameterization the uncentered one what does the gradient look like for the jacobian you can actually derive the formula for it which is the outer product of the training gradient with the current sampled hyper parameters and what this says is that moving the hyper parameters in the direction that they're currently in has the effect of moving the parameters in the direction of the training gradient right this is um perhaps wrong right this this comes right we want to update the weights of the network in the gradient direction but the jacobian is mistakenly believing that updating the hyper parameters in that direction will move the weights in the gradient direction and this looks like a really good direction to move in because that's the training gradient direction early in training it's probably well aligned with the validation gradient and so it thinks that it can get lower validation loss just by moving the hyper parameters in whatever is their current direction and so here's a toy example to illustrate this phenomenon we have a bi-level optimization problem shown on the left the outer objective is 1 10 times lambda squared plus w and the inner objective is w squared and this is kind of a trivial bi-level optimization problem because the inner objective doesn't depend on lambda so the response jacobian is zero and you can just solve it with simultaneous optimization of these two different objectives they don't even have to think about each other and if you use the centered parameterization like we see on the right that's actually what happens so starting from different initializations of lambda and w um the trajectories basically converge straight to the optimum for each of the variables independently but if you use the uncentered parameterization then what happens is that it mistakenly puts information that should go in w into the jacobian theta and so the jacobian changes along these trajectories and having a wrong jacobian means you have incorrect gradients for the outer objective and so so things are kind of pushed in weird directions um and i chose this example to illustrate the effect uh so it's it's sort of constructed to amplify this effect i don't think it's quite as strong during ordinary neural net training but this is basically what's going on and so just making this one change to architecture centering the parameterization can actually improve the results quite a bit so here we are adapting the l2 regularization parameter for linear regression and if we use the original stn we see that it initially takes a very large value before gradually going down to the optimal one whereas if we modify the objective by centering it then we can eliminate this pathology and converge to something resembling the optimal parameters much faster okay so that was the first trick centering the second trick behind the delta scn is to linearize the elementary network we want to approximate the network with the first order taylor approximation around the current weights similar to things like the neural tangent kernel so why is this a good idea well the problem is that when we sample lambda if we use a large perturbation scale then it can be fairly difficult to approximate the best response function as linear in the support of the hyperparameter distribution and so here we have an example of a bi-level optimization problem so we have the hyperparameter represented on the vertical axis and two inner parameters on the horizontal axis and the contours show the inner objective for any particular value of lambda and so in blue we're showing the exact best response function the dashboard line shows the exact best response jacobian that's the thing that we would like to approximate but if you actually optimize the linear hyper network with the distribution of hyper parameters you actually get what's shown in the red line that's the optimal linear hyper network and the approximate jacobian is the slope of the red line and so unfortunately the slope of the red line is very different from the slope of the blue line which means that our estimated jacobian is fairly inaccurate and so why does this happen well if you look at the cost for the for the true jacobian showing the dash blue line we can actually see it's much higher than the cost for the red line because if you move too far from the current hyperparameters uh shown in the star then it will fall far outside the contours so it has to be very conservative in how it reacts to lambda in order not in order for it not to be too non-linear and so why is the best response function so linear well i claim that it's because the neural nets function itself is non-linear in w and so that non-linearity is what causes the best response function to be non-linear and hard to approximate but if you think about why do we need this hyper network we need it for the jacobian and the jacobian is defined infinitesimal perturbations to w and so if we only need to consider infinitesimal perturbation we would be able to approximate the network computations is linear and so what we do in this approximation is we replace the network computations with their first order taylor approximation around the current weight vector so essentially the same approximation that leads to the neural tangent kernel and this can be written in the shorthand that delta y equals the jacobian times delta w because this is the origin of of our nerve of our term delta scn and so what are the effects of linearizing it in this way well there's the disadvantage which is that now we're approximating the network computations but the advantage is that now the response function for this linearized network is more linear and so we might be able to approximate it fairly accurately with the linear hyper network and so it seems that the second effect wins out and we can actually approximate the response to jacobian much more accurately with the linearized network so that's one interpretation we also derived an alternative interpretation you can find the details in our paper and basically you can interpret this whole training procedure as approximating the response jacobian for a modified inner objective um this inner objective is the rather than the regularized training loss with the current type of parameters we define the modified inner objective to be the expectation of the regularized training loss with respect to the perturbed hyper parameters right so we want a single weight vector that is fairly robust to the choice of hyperparameter values and so that so the best response function for this modified inner objective is the one shown in green and we're taking the jacobian to that that's represented as the dash green line this is a much better match to the exact best response jacobian in blue so even though we're we've modified the inner objective it's jacobian seems to be a pretty good match to the original one all right so how well does this work so we can evaluate these different procedures on linear regression problems where it's possible to find the exact optimal hyperparameter and so we're comparing both the choice of l2 regularizer on top and the dropout rate on the bottom and what we see is that the delta stn is consistently able to converge faster and get a more accurate approximation of the true optimal hyperparameter compared to the original stn moving beyond linear regression we can also consider training a deep linear neural network with jacobian norm regularization and so deep neural networks are interesting because they're equivalent to linear regression in terms of what they can represent and so there's a lot of analysis that we can do analytically about these networks so for instance we can the optimal regularization parameter for the jacobian norm just like we would for linear regression but the dynamics of these networks are much more like the dynamics of ordinary nonlinear neural networks and so we can separate out the original stn from the centered stn basically applying the first trick i mentioned the centering trick and then we can distinguish that from the full s for the fault delta scn that involves both the centered parameterization as well as the linearized network and we see that the centering helps and that the linearization consistently helps further we can compare the optimization of the validation loss on various problems and we see that for both an mnist mlp and a fashion mdis cnn that the delta stn consistently outperforms the original sdn in terms of the speed of minimizing the validation loss and so here we have comparisons on a variety of problems so we're comparing both the scn and the delta stn against the baselines of random search equation optimization and we see that the original sdn pretty consistently outperforms both random search and basic optimization and that the delta stn consistently outperforms the original stn in terms of the validation loss so to conclude i've introduced the self-tuning network which is another way of estimating hyper gradients so hyper gradients are almost always estimated either using unrolling of the inner optimization procedure or the implicit function theorem and so self-tuning networks are a third approach they're based on painting hyper networks that exploit the structure of the best response function so we can amortize the computation of the hyper gradients by training the hyper network and that allows us to approximate the response jacobian very cheaply just by differentiating through the hyper network so this whole thing is trained simultaneously as a simultaneous game so the hyperparameters are adapted online as network trains we also introduced the delta scn which uses a centered parameterization and a linearized network and this improves the speed and stability of the stn training and so this whole thing has a very low overhead compared to just training a single network so it requires the same amount of memory for the activations about twice the amount of memory for the parameters and depending on exactly how you set up training probably about a factor of two or three overhead in terms of the running time of the procedure and so this method is able to produce solutions competitive with patient optimization in a single rubber training so i think this opens up a lot of interesting future directions so the first one is a fairly natural extension which is to move beyond the iad setting of training set and validation set drawn from the same distribution and so we can consider settings where there is a more fundamental shift such as a change in the data distribution or some form of corruption that's only applied at validation time i think there's also a lot of interesting algorithmic work to do to understand the game dynamics induced by the simultaneous optimization of the hyper network and the hyper parameters it's a kind of simultaneous differential game but it has perhaps a lot of additional structure specific to the hyper network setting and it would be interesting to see how that can be exploited to improve the optimization performance i mentioned earlier that in adapting the hyperparameters we're only accounting for the explicit effects of regularizers like dropout it'd be very interesting to see if these methods can be adapted to account for implicit regularization effects as well and finally i described hyper network architectures that were suitable for the particular case of hyper parameter optimization but i think the general strategy that we've been taking of training hyper networks whose structure is motivated by a particular problem i think this can be applied more broadly to other situations where we need hyper gradients such as influence functions neural architecture search and related problems so i think that this work opens up a lot of exciting directions so thank you all for attending i'll open it up to questions okay thank you thank you roger as i said in the beginning i really love this work when it was just a pdf somewhere and it was really really great to hear about it um um so i guess we have i would be very efficient in our usage of time because we have five minutes and i have a bunch of questions um so the first one um is around what how would this work when you have hyper parameters that affect the capacity of the network so when you're changing the number of weights for instance um i i assume that there is a way to design an hyper network that that works in that case but yeah do you want to comment on that right so i think the challenge of these architectural hyper parameters probably has to do with the discrete aspect of it so the fact that you have a discrete number of units in the layer so you can't literally do gradient descent on it uh we think that you could use something like reinforce in those settings and you'd still be able to make use of the hyper network so you still have the kind of amortization of the response function so it might be like slower to adapt lambda but lambda is intended to adapt fairly quickly anyway in our experiments i guess there's a question of how important the response jacobian is for architectural hyperparameters because you could treat it as a kind of simultaneous optimization problem and only use the direct term in which case you basically be trading off the computational cost of running the network or some other resource metric versus the performance of the net network and um i guess approaches like like enas and darts and things like that um use some information in simultaneous um if i remember right um some of these methods use some information from the response jacobian but it's not clear how important it is or if it is really more like a simultaneous game so most of the hyper parameters that we've looked at were cases where pretty much all the important information is in the response term yeah it makes sense i guess i was thinking um the architecture search is one case but another case could be when you're just one you want to just want to change the number of channels of a convolution right uh do you think the responsibility would work well in that case or do you think that's i mean because it's one of the basic type of parameters you will want to tune right yeah i mean i think the intuition for what the response jacobian would do is say like if this unit is turned off then you should adjust these other units in this way in order to capture some of the same information and how useful that is um i don't really know right it might be that you just want to like order the features and if you drop some you just lose those features and you don't have to react to it in which case the bi-level part wouldn't be all that important right uh no that's that's really interesting um i guess a relate well not so related but but kind of i was um i knew about the work but i was very surprised by some of the results particularly for vision problems where you basically showed like strong online performance you basically don't have to uh train from scratch again after you find the optimal lambda star in some sense um uh so do you have a sense of what you know where does that performance come from does it come from the schedule so to speak does it come from the fact that you learn a function really or does it come from the optimum so if you try and retrain just from the optimum does that achieve the same performance or better performance yeah so um in at least some of our experiments we did retrain from the optimum and that also performed about as well and i don't remember which subset of the experiments we did that for we definitely did that for the the dropout schedule experiment they mentioned um so i think there's a lot of issues in the kind of path dependence that we don't really understand because um as the like hyper parameters adapt then as the network is optimizing to the current hyper parameters it still has some memory of the previous hyper parameters and so there could be useful information that was learned in those that it wouldn't be recovered if you started again from scratch so they're definitely like those sorts of effects are definitely possible in principle and understanding how this whole procedure interacts with the implicit regulation and normal minimization and things like that seems like a really important problem and so so it seems like it didn't hurt us too much in practice but we don't have a good theoretical understanding of why yeah that's great i'm watching time i mean this is a fascinating discussion because i do think that for augmentation protocols it could be super useful and you you do have an augmentation with you saturation and brightness uh anyway i wish we could talk about this more i guess we'll talk more offline uh i wanted to thank you so much for presenting this as i said this is fascinating work and thank you so much for sticking around and answering all the questions in the chat as well there are more coming in in front of my eyes so apologies to for all the questions we couldn't answer and thank you again roger all right see you bye