Synthetic Gradients Explained

Key Takeaways

hello world it's Suraj and deep learning it works so well why does it work so well the answer is because it uses a technique called back propagation and in this video I'm going to demo a newer strategy that optimizes neural networks that is better than that propagation and it's called synthetic gradients this technique was released by deep mind a few months ago and it just hasn't had enough attention paid to it this is what happens in research a lot right some great ideas don't get enough attention to pay pay to them so then some youtuber like me has to make a video about them so what we're gonna do is we're going to take the simple feed-forward neural network it's a simple neural network that learns to predict the output of adding two binary numbers together and first we're gonna train it using back propagation and so I'm gonna go over back propagation and then I'm going to change the code so instead it uses synthetic gradients so this gif that you're looking at right here is an example of the synthetic gradient strategy the idea behind synthetic gradients is rather than waiting for your neural network to go through a full forward and backward pass you can just update each layer without having to wait for that using what's called a synthetic gradient these are gradients that are not the real gradient the true gradient but instead an approximation of what that gradient would be so it's a synthetic gradient and by doing this we can update each layer without having to wait for all the other layers to update it's a faster way to optimize neural networks and it allows us to decouple each layer from having to wait for all the other layers so let's start off by talking about how neural networks learn in a black box approach the neural network gets an input that's the input data and some desired output right so it's got an input in a desired output and it uses that to update its internal state accordingly right we have some desired output those are our labels right all of all of that propagation centers around having some labels for our data we've got to have the labels if you don't the labels you got to get the labels right and so if we have our labels we can train our model and so once we've trained it then we can perform prediction and the prediction process is using just the input and the internal state to generate the most likely output according to its past training experiences so we have some input training data we feed it to this learning system which is a neural network and it Purdue and it predicts an output we then compare that predicted output to the actual output and we compute some error value we use that error to then update the learning system so then every time it gets better and better that's just a training process and that updating process is that propagation so in a more detailed sense we have some neural network right we've got an input layer we've got a hidden layer and we have an output layer and Egypt so each of these are layers right these three circles are a layer these three circles are a layer these last two circles are a layer and each of these circles represents a number because we are inputting let's just say three integers right a 1 1 and a 0 that's the input and the reason that there are these black lines is because we are multiplying the input we are multiplying the layer by the next layer so we are multiplying this one value by all of these other values in the next layer because a neural network is input times weight add a bias and then apply a non-linearity to it so that's what we just keep doing for a neural network we take our input data we multiply it by a weight matrix which is the next layer we add some bias value and then we apply a non-linearity to it which is a sigmoid function or there are there are plenty of nonlinearities that we can apply and this is the this is the this is called forward propagation this is how we take some input data and we just apply a series of operations to that input data until we get an output prediction that we denote here as Y hat and so that is the whole forward propagation process a neural network is just a series of operations that are applied to some input data right that's it input times weight add a bias activate repeat that process over and over and over again until we get that output prediction and once we have that output prediction we could compare it to the actual output right and so in a very very basic case we can just subtract the actual output from the predicted output and that difference is the error right we what we want is that error to be zero we don't want there to be any difference we want our predicted output to be the exact same as our actual output but as long as there is an error we want to minimize it so it gets smaller and smaller over time that is the training process and so how do we minimize that error well once we compute the error we can compute what's called the partial derivative with respect to each layer going backwards right so we compute the partial derivative of the error with respect to the weights for the previous layer and what that's gonna do is that's going to give us a gradient value then we take that gradient and we compute the partial derivative of that gradient with respect to the weights of the next layer or the layer before that and we get another gradient and then we take that gradient and we compute the partial derivative of the gradient with respect to the layer before that and so we just keep doing that and so if we keep doing that we get great values for every single layer and then once we've done that for every single layer then we could use those gradients to update the weights of each layer so you might be asking what is a gradient a gradient is a direction it's a direction on how to update our weights and it comes from calculus calculus is a study of how things change that's why it's used in physics right how do planetary bodies move over time how do the weight matrices of a neural network change as we are trying to minimize an error the gradient the gradient is a value it's a value with a direction and what I mean by Direction is if we were to if we were to graph all the possible weight values that we could have for a single layer versus all the possible error values that we could have for that same layer it would look like a parabola and what the gradient is is it is this it is it is it is synonymous with the partial derivative and the partial derivative or the derivative is the slope of the tangent line to a curve at a specific right so that's what the gradient gives us it gives us that slope of a tangent line it's anonymous and the direction that that slope is is pointing tells us what direction we need to move such that we can get to this point where the error is the smallest that is the minimum of the parabola and once we get to that point then we know immediately what the ideal weight value should be so the error is smallest and we do this for every single layer right we compute this gradient value for every single layer and the training process process is us generally it is us gradually decreasing the point that we are at for each weight matrix until we are at this point where the error is smallest and all the weight values of that weight matrix are equal to when the error is smallest and that is when we are finished training that is the optimization process and because we are descending the parabola it's called gradient descent so great descent is a very popular optimization strategy it's used in neural networks it's used all over the place but in the context of neural networks we call it back propagation because we are back propagating an error gradient across each layer recursively so every layer depends on us having computed the gradient for every single layer before that right so every layer is coupled it's dependent on us having computed the gradients for every single layer that happened for every single layer after that so what this looks like in code is if we were to look at this in numpy is if we were to look at this in Python what I've got here is a function called generate data set what this function is gonna do is it generates a bunch of binary numbers right like 1 0 0 1 and then 1 1 1 0 it's also going to return the outputs and those outputs are the sums of those binary numbers so if we take two binary numbers like 1 0 0 1 0 and then we add another binary number like 1 0 0 1 it's gonna have some some value that is also a binary number like 1 0 0 1 the input for our neural network are going to be these two binary numbers concatenated together so there are one big number and the predict and the actual output will be the sum of those two binary numbers so we already know what the sum is going to be right we've we've already computed it in this function but what our neural networks job is is to learn what that output would be so that even if we don't have that output we can predict what the output would be so it's making a prediction by updating its weights and going through this back propagation process without even having to add the numbers together it's just able to predict what the output would be so what we have here are this the sigmoid is the non-linearity and the reason we apply a non-linearity to neural networks is because neural networks are universal function approximate errs they can compute any function whether it's linear like a like a straight line or nonlinear like a very curvy line like a parabola they can they can they can approximate any function and that's why we use a non-linearity so they can approximate both linear and nonlinear functions this is the this function helps us compute the partial derivative with respect to each weight going backwards right so it's the it's the derivative of the sigmoid function and that's what's going to help us compute back propagation so this class is a is a layer class each layer is initialized by its input dimensions and its output dimensions which are the the dimensions of the numbers going into it and out of it it's got a non-linearity which is our sigmoid and it's got a non-linearity derivative which helps us compute the gradient value so each layer is initialized as a weight matrix of some size that we define here it's also got a non-linearity and a non-linearity derivative right that's how we initialize a layer it's a glorified weight matrix then we've got a forward and backward a forward and backward function the forward function takes the input it's given and it takes it takes the input multiplies it by the weight multiply meaning the dot product right when we multiply two matrices together it's considered the dot product that's why we use that's why it's considered linear algebra and not regular algebra the dot products comes from linear algebra linear algebra tells us how to multiply groups of numbers together rather than just single integers which is what algebra helps us do that's what we use linear algebra in deep learning so we do the input times the weight and we apply an activation or a non-linearity to it input times weight add a bias we're not having a bias right now for this simple example and then activate right and that's going to give us our output so we repeatedly apply this forward this forward function and that's going to compute our forward propagation and then for the backward step the Delta is the gradient value for the backward step we take the partial derivative of the output value which is the predicted output and we multiply it by the gradient right and that's going to give us the the the gradient value for that weight and then we return that and then one more function which is the update function which says okay you've got your Delta or your gradient value multiply it by the input and that's gonna and then and subtract it from our weight values and multiply that by an alpha value which is the learning rate and that's going to give us the updated weight after we computed our backward propagation process okay so we start off by saying we don't want this we want this to be a deterministic algorithm so it's gonna compute the same results every time which is useful for debugging we've got our number of examples or output dimensions how many times do we want to train a thousand and then we generate our data set right using the function we define earlier for x and y then we say okay we've got our three layers and we define them as such right we have our dimensions for the size of each we have our non-linearity and we have our non-linearity derivative those are our three layers then for our training process we're gonna say okay here are our batches of our x and y our training data and our our training input data and our output data as well right the two binary numbers concatenated together would be a batch or a set of our training data and the Y would be the expected output right so what we do is we forward propagate taking that first input data as the input to get the output value and then we forward it forward propagate that output as the input the next layer and that's gonna give us the output for that layer and then we take the puppet for that layer and we feed it as the input for the next layer that's forward propagation and so now we have this output prediction we subtract it from the expect expected outputs and that's going to give us the error value and we use that error value to then as an input to the backward propagation for the layer before that to give us the gradients values right the Delta the change and it's using the partial derivative with respect to the weights for the previous layer to compute the gradient and we do that recursively every time and once we have all of those gradients then we can update every single layer and then we are done with that and so if I compile this notice how the loss decreases every time and as the number of iterations increased the loss is slowly and slowly and slowly getting smaller and smaller okay so that is the back propagation process so okay so now let's talk about the problem with back propagation so back propagation is how all of supervised deep-learning works the majority of deep learning when we have labels for our data but there is a problem here so check this out this is an example of that propagation right here a very simple example of a three layer neural network so a layer can only be updated after a full forward and a backward pass has been made so after layer the first layer has processed an input it updates after the output activations these black lines right here have been propagated through the rest of the network generate a loss and the error gradients these green lines have back propagated through every single layer and what this means is that layer 1 must wait for a full forward and backward pass of every single layer that comes after that so layer 1 is locked it's coupled to the rest of the network so for this simple example right here or the one that I just demoed programmatically it's not a big deal but when we get to more complex neural networks that's where things start to get hairy right for example the inception network it's a huge convolutional network with 19 layers or the differentiable neural computer with its separate memory unit or you have the neural machine or you've got bi-directional recurrent networks just insanely complicated neural networks with all sorts of moving parts when we get to more complex systems the time it takes to have to wait for the gradient to be back propagated through every layer is inefficient ideally we don't have to wait right so over a big distributed network over multiple machines this could take a very long time having to wait for every layer so if we decouple every layer the connections between them so they can update independently without having to wait for the other layers to update that would be more efficient right and so that's the idea behind synthetic gradients right so normally a neural network compares its predictions with data set to decide how to update its weights right it uses back propagation to figure out how each weight should move in order to make a prediction but with synthetic gradients individual layers make a best guess an approximation of what the gradient would be it's not the actual gradient it's a synthetic gradient a fake gradient a predicted gradient and so this best guess that each layer will make in this synthetic radiant model is called the synthetic gradient and the data is only used to help update each layers Gesser so how are these synthetic gradients generated I'm gonna talk about that in a second but what this does the system of approximating gradients at every layer it allows individual layers to learn in isolation without having to wait for the other layers which increases the speed of training so this is kind of what it looks like right here so we have some input right we have some input data we apply it to the first layer so input times weight add a bias activate and then we send the output to the next layer then rather than waiting for that for that data to forward propagate through every single layer compute the error and then backward propagate the gradients again again and again until we get that gradient and having to wait for that full forward and backward pass before we can update it we just take the input apply we just take the input time's wait a Tobias activate send it out and then we use what's called this synthetic gradient generator to immediately update the weights for that network so we take an input compute an output and then update it without having to wait for full forward and backward pass and we do that for every single layer so you might be wondering okay how is this possible right this give how how are you able to generate a synthetic gradient without having how are you able to do that well the answer the answer is the synthetic gradient generator is in fact another neural network it's another neural network it's a very very simple single layer network right they deepmind found that even when they use a single layer linear layer as a neural network for the generator it still works really well right so it doesn't have to be some complex neural network but basically the synthetic gradient generator is a very simple neural network that's trained to take the output of a layer and predict what the gradient would be the likely gradient at that layer it's simply a neural network that predicts what the gradient would be when we perform a full forward and backward pass then we get the correct gradient we can compare this to our synthetic gradient so we can train our synthetic gradient networks by pretending that our true gradients are coming from this mythical data set so if we look at this image right here notice how the gradient for M I plus two back propagates through F 1 plus F I plus 1 and into m i+ my plus 1 so each synthetic gradient generator is actually only trained using the synthetic gradients generated from the next layer so for every layer it doesn't depend on every single layer before that right because that propagation is a recursive process rather than having to wait for the gradient to be computed for every single layer it's only taking the it's only taking the computed gradient from the next layer every layer only depends on the next layer that's it so the the synthetic gradient generator is a neural network and it it generate a synthetic gradient and it compares that synthetic gradient to the actual gradient and actual gradient is a gradient that's computed from the next layer and it computes an error value using that and then it updates its weights using that and it's and it sends the synthetically generated gradient to that layer and that's what updates the layer so every single layer is using the synthetic gradient generator to update its weights only the last layer Jen updates its weights using the true gradient value because there is no next gradient it's using the true output label to compute its gradient and so what's happening is because every layer is dependent on the next layer just-just-just on the next layer to compute its synthetic gradient the what happens is it doesn't magically allow for a neural network to Train without having that the true grain back propagated the true gradient is indeed still percolating backwards it's just slower do you see what I'm saying so the the true gradient is slowly percolating backwards not like the real gradient every time right because only the last layer is computing the output using the true label of the data but every other layer only uses the only updates its weights using the its own its own respective synthetic gradient generator which uses the gradient from the next layer as its as its true gradient versus its synthetic radians right so the true gradient is slowly propagated backwards however because we're using the synthetic gradient model to approximate and smooth over the absence of true gradients it's faster and so they they found that this could be applied to any type of neural network architecture not just feed-forward networks recurrent networks it's an awesome strategy I would like to see more of it integrated into major deep learning libraries and it allows for the training of distributed networks to be faster and cleaner so what I want to do is I want to update our code so that it's now using the synthetic grade model rather than the back propagation model I'll name this class the decoupled neural interface instead of a lay okay and so I'm gonna say so this stuff stays the same as before this doesn't change at all these weights the non-linearity and the non-linearity derivative they don't change but what changes is this we have some new stuff here we have some new stuff and that's gonna be the weights of the synthetic gradient generator which we can initialize randomly just like we would for any weight matrix for a regular neural network and it's gonna be generated using this output dimension size just like that just like that and we have it's it's gonna have its own learning rate right so this is the synthetic gradient generator which is its own neural network so now we want to update this forward propagation step right so to update this forward propagation step here's what its gonna look like when I update it for synthetic gradients okay so this used to be just the forward set but now we can update during the forward pass using synthetic gradients so we have some cash input we forward propagate that input and then we generate a synthetic gradient via simple linear transformation right so we use that weight matrix to compute the dot product with the output and then we get our synthetic gradient right input times weight matrix no bias then we can update our regular weights using the synthetic gradient so we take the partial derivative of the output times a synthetic gradient and that's going to give us the weight update that we can apply to our real weights for that layer it's called weight synthetic gradient and the way we apply it is by multiplying the input times the transpose of the transpose of the dot product of that synthetic gradient synthetic gradient multiplied by a learning rate and that's how we update the weights for that layer and finally we can return the back propagated synthetic gradient and the forward propagated output so we're returning the back propagated synthetic gradient and we are returning the forward propagated output at the same time right so each layer gets an input it for propagates that input and at the same time it's up with a synthetic gradient value which is very cool and so in this case we don't even need this backwards step so we're gonna replace the both of backward in the update step with this new update synthetic weights method where we this is just like the update method from before except it operates on the synthetic weights right so we are updating the synthetic gradient generators weights that separate neural network we've already updated our real layers weights now we want to update the synthetic gradient ways right so what we do is we say we take the synthetic gradient minus the true gradient which is the gradient from the next layer and we use that to compute the Delta the change and then we use that Delta to update the weights of our synthetic gradient generator so then what is the difference here well the difference now is instead of using each the layer class to define these layers will use the decoupled neural interface class well we have our input dimensions as for the size of the input the sigmoid and this the the activation function and then the derivative and then the learning rate and that's how we define every single layer generating our batches say this stays the same but the forward and the backward pass are changed entirely rather than it being a separate forward and a backward pass we have instead a forward and synthetic update pass and then we can update the synthetic gradient generators after that so let me just move this out of the way right so we have for every layer we're doing our where for propagating the input data from that layer and at the same time we are updating its weights using the synthetic radians when we do that for every single layer okay and so then once we have that we want to update the synthetic gradient generators weights so we we say okay we know what the real output is going to be so this layer 3 Delta is the true gray and we only compute that for the last layer every other layer is updated using its synthetic gradient generator its respective synthetic gradient generator and then and only then do we update so that's the only the last layer is is normally updated it doesn't have its own synthetic gradient generator but every other layer does and so we use that gradient to then update the synthetic weights for each of the generators backwards right and so that's going to and so right and so that's how synthetic gradients work in a nutshell and I think it's a really cool idea I'd like to see more people looking at it I'd like to see more of it applied in deep learning tensor flow all the major libraries should should have synthetic gradients as a valid optimization scheme and I think we're gonna see more of that if you want to learn more about artificial intelligence about blockchain about some of the coolest technology out there hit the subscribe button for now I've got to go define proof of proof of proof so thanks for watching