Modeling Evolution with Tensorflow.js

Key Takeaways

hello world it's Suraj and evolution it's the reason we're all here today in today's video we're going to use tensor flow dot J s to simulate the process of evolution in our browser what you're seeing behind me is the demo that we're going to code in today's video what it's doing is it's using a bunch of different neural networks and it's using evolutionary strategies to evolve all of these neural networks over time to learn how to walk from the left end of the screen to the right end of the screen so every generation a new breed or new generation in every generation a new generation of neural networks are going to be bred and then they're all going to try to walk from the left side of the screen to the right side so now we're at generation 7 they're getting a little better they're still kind of jumpy and they don't really move that much we can take one and you can't even like move it across like boo alright it's gone but yeah it's using tensorflow touch a you has to do this and it and evolutionary algorithms it's the combination of neural networks and evolutionary techniques put together that can allow us to evolve a population over time and this this has some real use cases and I'm going to talk about those in today's video we're gonna code this in tensorflow j/s at the end of the video but I want to start off by talking about neuro evolution ok so machine learning models neural networks in particular neural networks are function approximator that means they're really really really good at approximating the correlation between some input data and some output data right so the input data could be images they could be videos they could be numbers and the output data could be labels they could be you know hot dog not hot dog they could be 1 through 10 they could be the price of a stock on a given day they could be the note for a musical composition whatever but it's the label and there is some mapping between the input and the output data and neural networks are really good at learning that mapping and the way we do this is using a technique called gradient descent okay so gradient descent is all about finding those optimal values by finding what's called the minimum in some valley right so if we were to map all of the parameters versus the output values you'd find some valley and we want to find the bottom of that point right so it looks like this and what gradient descent does it's iteratively in every time step it tries to find this minimum value and once it finds that minimum value we could say okay we've got this minimum point what is the what is the coefficient values at this point so in this three in this three axis model we have a z axis a y axis and an x axis so in the X and y axis represent the two parameters of a model and the z axis is the objective it's it's that output value and we can try and guess what those I use are but but by using gradient descent we can learn what the optimal values are and the way to do this is write gradient descent that's how it's done so far but in the 80s a couple of researchers thought okay what if we use a different optimization technique what if we don't use gradient descent and if we think about life on earth you me animals bacteria we're all neural networks we all have these networks of neurons in our head and we're learning through trial and error and the the local learning procedure could be whatever right we don't know what it is but the more macro learning procedure through our children and their children and their children is this process of evolution right so we are adapting to our climate we're adapting to ours our scenery we're adapting to our environment the context of where we live and we pass on these adaptions in the form of genes to our children and that process is evolution and that in and of itself is a learning technique and what evolutionary algorithms neural evolution does is it tries to model that process in simulation in silico inside of a machine and so if we think about a bunch of different neural networks we can randomly initialize all of them so they all start out with random weight values and what we can do is we can pick the best one by some measure of whatever best means for our objective and then we can take that best neural network out of say 100 and then say okay this neural network is the best of all of them let's let's create a bunch of other neural networks that are slight variations of this best one and that's the new population and then out of those there's gonna be a best one and we'll take that and we'll breed it with another and then we'll repeat that process over and over again so we're just kind of simulating evolution that's so high-level I'm gonna go into the details of how that works second but what I want to say here what my point is that my point is that gradient descent works really well if you look at all the production systems gradient descent is the optimization strategy also called back propagation in the context of neural networks it is the optimization strategy that is used however with uber with a bunch of new startups and uber and some big companies I'll talk about more and more companies users startups researchers are using evolutionary strategies neural evolution genetic algorithms to optimize their neural networks and I'll talk about those in a second ok so think about it gradient descent is us trying to find that minimum point by starting at some point and then an eerily going closer and closer to that ideal value if you want to look if you want to know how back propagation works how gradient descent works search back propagation and Suraj on YouTube back propagation in five minutes video it's super useful okay so so it typically goes like this like I said we will generate a population of let's say a hundred random neural networks right so we don't know what those initial weight values are so in the context of neural networks those parameters would be weight values right so you know this is initialized with some random you know group of numbers a vector of weight values another one is with another group of weight values and we will then determine the arc and all those architectures are determined using some distribution right so some Gaussian distribution etcetera then we'll set to select the best neural network by some fitness function we'll talk about that well fine let's say three or four of them will breed them together by breeding it means you know it could be a dot product between those weight values it could be finding those optimal parameters and and then choosing those for the next one etc what those weight values are copying them and then we'll create these offspring which are new neural networks and then we'll repeat that process over and over and over again so there are a couple of genetic algorithms out there there's neat neural evolution of augmented topologies there's hyper neat there's novelty search there are quite a lot of examples and so the idea here is that we start off with some neural networks and then we evaluate all their how fit each of them are so that means checking where the models are in the optimization surface which of the models perform best and then selection is performed based on that Fitness evaluation will select four or five whatever threshold value we decide in evolutionary strategies the offspring are is reduced to a single model and weighted by the Fitness evaluation so for deep neural networks the fitness is defined as the loss or the reward and essentially we're moving around the optimization surface and using the offspring to get in the right direction so that the the key difference between gradient descent here is that instead of computing the gradients we're setting out multiple antennas and moving in the direction that looks best there's not a single model that we're optimizing there are several there's a population of models that's that's the big difference here right so if you think about there's there's six of these models here think of each of them as as all of these rows each model one model equals one row we'll select the best ones we'll combine those doggies using some kind of technique maybe dot product multiplication division whatever then we'll mutate them that means we'll vary them in some way and then we'll place those mutated offspring in back into the population and then try and try again the process repeats over and over again in this way it's similar to a structured random search but the end result is in the selection phase that we have a single model and then we you know repeat that process over and over and over and over again so this is an example of uber so this is this map I took from Hoover right so they're engineering a website so they having this great blog on neural evolution which I have a link to but basically what one technique they used one one real-life use case was to use evolutionary strategies to evolve this pathfinding algorithm to get from the best route from point A to point B and this is the map of what they found to be that that best route [Music] and so so applied to neural network specifically we can use genetic algorithms to optimize any kind of model it doesn't have to be a neural network it could be you know some simple function it could be some it could be anything really I know neural network really is just a nested function and every layer is another layer of nesting that function you know f of X is one layer and then M of f of X is two layers and then G of M of f of X is a three layer network you see what I'm saying because we're taking the output of one function and using it as input to the next function and using the output of that function is it and using it an input to the next function so it's it's it's this it's this nested function right so however many layers is another layer of nesting so applied to a neural network if we apply neural evolution to that so what is what is that thing that we want to optimize what we want to optimize the number of layers the neurons per layer the activation function the network optimizer these are all different parameters that we could optimize and so when it comes to use cases it really just boils down to anything that could use a neural network and that could improve on the back propagation optimization strategy so uber found actually in their blog was that neural evolution outperformed great descent in terms of time to convergence so they sped up this Atari game this this deep Q learner that you know to deep mind like days and days on massive sets of GPUs to compute into a single hour on a desktop CPU which is really cool so just check that out I have a link to it in the video description but so like I said it could be applied to anything that neural networks are currently applied to if there is some if there is some AI that is using a back propagation right now that is slow in some way where a 2% 3% 5% 10% increase can be useful that would be a perfect time to apply this technique to write you can outperform the competition by using an evolutionary strategy right so this could be applied to self-driving cars better game AI image classifiers like clarify art generation really anything that's using neural networks so you can apply a neural evolution to to make that problem ization process faster that's what it comes down to a faster optimization process that converges through a better minimum faster right so you can get better results and faster if you are using neat algorithms most of the time from what we've seen here in some of these examples especially the uber blog but also Google so Google users use it for auto ml their automatic machine learning algorithm that you can try in the browser and deepmind used it recently so deep minded neural evolution for pop they called it population-based training I remember it was like population based training population based training of neural networks that's what they called it yes and this was released less than a year ago but yeah some great blog posts on that as well they've got some nice little visualizations of using them on generative adversarial networks great use case popular model so yeah there's a ton of use cases but in today's video I want to talk about using it in the context of tensorflow das this is the hottest machine learning library right now because it allows us to create ml models in the browser that anybody can access on any kind of device mobile embedded device TV laptop desktop whatever and it's it's really utilizing the power of transfer learning which we haven't really seen when it comes to Python or C++ for some reason it's people are using pre trained models much more and I think it's because it's a lot easier for one and two it's using JavaScript which has a much bigger user base right there are more programmers using javascript these days so before we get to our model I just want to say two quick things about tensorflow that Jay has to remember there is the core API right so the core API is what we're going to use button lets us build a computation graph and we could think of a computation graph it has a neural network rights because it's just a series of operations that tensors data flows between and these operations could be layers they could be you know whatever you know different operations like adding subtracting depending on the type of architecture you want but it's very simple to do in JavaScript here's a simple example we have two tensors a and B we add them together we print them out just like that and that those are two nodes in the computation graph a B plus that output that's the computation graph three nodes in the graph a B and then the total the output of a and B so and that's that's what we're going to use because this is a very simple neural network that we're going to build however when it comes to more complex neural networks with much more lay many more layers and you know many different you know strategies like attention momentum etc then we want to use the high level API which is the layers API so layers API lets us it packages together both the variable and the operations that act on Tim and act on them into a single function so one example would be this so that the layers the dense line of code it performs a weighted sum across all inputs of each output and applies an optional activation function so it's combining but the variable and the operation into a single line which doesn't seem like that big a deal but when you're creating a big neural network it saves a lot of time it's basically chaos in the browser that's what it is think of that as an analogy chaos in the browser a single line per layer okay so now let's get to car models so in our model what we're gonna have are these creatures and each of these creatures is a three layer feed-forward neural network that's their brain that we see right here right there's a bunch of creatures right now in this generation there's only one but if i refresh we're gonna we're gonna get a lot more that's a lot of creatures right so each of them has their own neural network so the input data that's fed into the network are four different parameters the horizontal velocity how fats are moving horizontally the vertical velocity of us moving up and down the torque how fast they're turning and the height above the ground level so the score the way that we are scoring these these creatures is we're saying they can gain points based on the distance that it travels from the starting points so the further it travels in the correct direction the more points it gains so if it goes from left all the way to the right it's gonna get a lot of points and then traveling in the opposite direction is going to reduce the point so the fitness function the way we define how fit a neural network is in this context is that how far a creature goes the selection algorithm that says let's see what are the best most fit neural networks here is to select based on their fitness value which which will be a scaler a single value and that's going to be that's that Fitness value is going to act like the probability of being chosen for reproduction the creatures that perform better have higher Fitness values and hence have a higher chance of reproducing when it comes to crossover breeding reproduction whatever you want to call it two creatures these parents are selected using this selection algorithm their weights are interchanged randomly in our case and then finally mutation is a single value a parameter that's going to change the probability of introducing randomness to each of these networks so let's get into the code here okay so I want to start off by just creating a simple neural network this will be in 40 lines of code we're gonna use tensorflow j s so I'm gonna call this a neural network class let me make sure that it's using JavaScript as its base javascript there we go okay so it's gonna be called class neural network okay and in this classroom Network I want there to be a constructor the constructor will define what the input to this neural network will be we want some certain number of input nodes we want a certain number of hidden nodes or neurons and we want a certain set of output nodes okay so then inside of the constructor we're gonna define what those input nodes are what those hidden nodes are and that's gonna make them accessible inside of the class right so for all three of them I'm going to do the same thing and so these values are gonna be accessible later on in the class as I manipulate them with inside of other functions the next step is I'm going to initialize random waits for that neural network right because when our creatures start out they're going to be randomly initialized weights right because there's no learning happening yet and so now I'm going to actually use tens for SAS and like I said before we're gonna use we're gonna we're going to generate these weight values randomly from a distribution a Gaussian distribution and so to do this we can use the what's called the random normal function so from a normal distribution we're gonna generate a set of weights and the way to do this is to say well how many input notes do we have and then how many hidden nodes do we have and then based on those values we can generate a set of weights and then once we have that we could say well that's our input weights and now we want to have one more set of weights right output weights because this is a a three-layer like I said before a neural network and then between each of those three layers is a set of weights and because there are three layers there are then two sets of weights and so because there are two sets of weights we'll create those two sets of weights and because I had the input nodes and then hidden knows for the first I'll have the hidden nodes and output nodes for the next set of weights okay so now we've initialized our random weight values and now that we've done that we can create the predict functions right so we have our neural network it's constructed and we want to create a predict function right so that what the user will do is they will input some data right so this is going to be some input data and we'll talk about what that input data is but it's going to take that input data and we're going to define some output variable so now I'm going to use the tidy function tetravaal is a tidy function and what this is going to do is it's going to I'm going to use this as a wrapper and then inside of that wrapper I'm going to actually create my neural network so before what I did was I initialize those values and now I'm actually going to create the neural network itself using those values I created before what tiny does is it executes a function and it frees up any GPU memory so because we're about to create this network I'm gonna use the tidy function to execute this neural network and as a function and then free up any GPU memory that I have because this is going to require some GPU memory right now so this is gonna this is going to take an a 1d array as input so for the first layer I'm gonna say well I'm gonna input a tensor using and then that tensor is going to be created using what the input data is from the user right and the size of that is going to be the input nodes size that's the first part the next part is for us to say well what's the hidden layer the hidden layer is going to be the input layer but we're gonna matrix multiply it by the the set of weight values that we had that we define before right so inside of the constructor and we're gonna apply the sigmoid function to that because input times weight activate that's going to activate that result if you want to know why I'm using the sigmoid function watch my video build a neural network in four minutes now that we have that we can just we can use we can create the output layer because this is a three layer feed-forward neural network and I'm gonna do the same exact thing because neural networks are giant nested functions I'm going to take the output of the first function and use it as input to this function and I'll say this thought output Waits got sigmoid and lastly now I have my output which is going to be the output from the output layer data sink which is going to sink the result into a single scalar value that can I can have as my output and now I can return that output so that's it for the predict function now I have two more functions that I want to create so the first function is called clone so what the clone function is doing is it's gonna say let's create a clone e and what it's gonna do is it's going to create a clone of our neural network because we'll want that at the start we want to create a bunch of different neural networks right and so this function is going to do this we'll just call this function over and over and over over again and so whenever a population is initialized we'll cloth will call this function how for however many neural networks we want there to be and now that we've defined the neural network class we can just say create a new neural network use you know whatever number of input nodes you want whatever number of hidden nodes what you want and then whatever number of output nodes you want once we have that we'll say well now we can dispose of anything in memory that we don't want so we can clear up some memory and then we will say here are the input waits clone the ones that we already have using the TF Tech clone function and here are D output Waits okay so now that we've done that we have our input ways we have our output ways we've cloned our neural network and now we can return it and lastly now that we have all of that we can finally just say dispose which will dispose the input and help of weights from memory which we'll call at the end of a generation and once we have that we are done with our neural network class now we can look at the rest of the code you know just copy paste data and then this is going to be output okay so that's our simple class say that so in the context of our genetic algorithm we have a genetic algorithm class and this is just an example of how it works will initialize the creature the creature will be a neural network will define what it looks like inside of the space like the visualization by these these values right here and then will define some way of picking one of those one of those neural networks or creatures by defining some threshold value and if it's over that threshold value that is our selected that is our selected network and will return that now we can say it let's evolve our neural network so we'll assign a fitness to each creature we'll pick the best one and then we'll breed them by picking two parents does it that define a child and using this crossover function do to define what the child's weights are and then push that child to a new generation and repeat that process for all of those values with our children inside of that generation hooray we'll kill the current generation iteratively and then add new children to the generation population so this is the genetic algorithm class as a whole is said the neural network class as a whole and now we can look at how this is combined in sketch Jas so this is how it works we define the canvas what everything looks like well initialize a generation a person or creature we can call it whatever and we'll say let's define a bunch of them we'll add it to the world will define the world's boundaries the mouse constraint this is just visualization stuff in JavaScript and we'll say well let's restart a generation every five seconds now when we'll have a counter for that will display the stats for that okay and so the real the real the meat of this here is it just initializes population right here and we'll just say just keep doing that over and over and over again so generation don't evolve that's it for today's video I hope you found it useful if you want any more cool information it's gonna be in the video description I've got the code help for links everything for you check it out if you want to evolve to the next level in terms of your programming expertise and your life hit the subscribe button for now I'm gonna involve myself so thanks for watching