GameGAN Explained!

Key Takeaways

this video will explain the gamegan neural game engine from researchers at nvidia this model compresses the pac-man game engine into a generative neural network such that you can play pac-man inside of this learned world model game gan is incredibly exciting for the future of gaming for example this could support transferring a given game from one operating system to the other without rewriting any code additionally once this is further developed it could dramatically compress games by storing them inside of neural networks i'm also really excited about the potential of these techniques and this carefully designed architecture for model-based reinforcement learning and applications like robotic manipulation this video will explore this architecture of gamegan gamegan is carefully designed with three separate components a dynamics engine an external memory module and a rendering engine each of these components has significant detail in the design this model is trained with four losses three adversarial losses enforce realism of the generated frames that the generator is using the action input from the user such as shifting the pac-man to the left when the user moves left and ensuring the sequences of frames are temporarily coherent the fourth non-adversarial loss makes the network use the memory module to store static background elements and disentangles the static and dynamic elements of the game this video will look at the results of gamegan on pacman and viz doom and the success of training an rl agent in this model and transferring it to the real pacman game as well as the novel come back home task presented in this paper this video will explain the new game gan from researchers at nvidia this is a neural game engine that lets you play pac-man inside of a learned world model in addition to being cool for playing games this has huge applications for model-based reinforcement learning the game engine contains the logic of the game such as if else rules like the pac-man can't move down through a wall or left or right out of the borders of the game also rules like if the pac-man runs over a food it eats the food and gains some reward and if it runs into a ghost it loses a life and is set back to that initial point this game engine governs the rules of this game in more complex games containing things like physics equations of collisions and gravity so the idea here is to distill all this logic in the game engine into a generative neural network so we're compressing all this game engine into this generator neural network that just takes an action from a user has the history of the frames and then generates the next frame so the user can play the game entirely inside of this imagination or this neural network it's also really interesting because then the programmer doesn't have to write different versions of the game for different operating systems or machines because you can just run it in this neural network in addition to playing games a lot of these world models are useful for model based reinforced learning where learning these transitions directly such as knowing the state action next state transition is really useful for sample efficiency a lot of these techniques plan in a latent space like mu0 planet dreamer or plan to explore what this means is that instead of constructing the entire pixel space from the compressed representation we just plan the future in these compressed vector spaces so say going from an 84 by 84 by 3 rgb image into something like a 100 by 1 vector and we would plan in these vectors rather than the pixel space so this game gan is going to construct the pixel space another really interesting paper that does this is world models this blog post from researchers at uc berkeley described their efforts to build robots that can manipulate fabric like folding towels or t-shirts in their research they describe the benefits of model-based learning we show that deep model based methods have more potential for generalization to a variety of tasks provided that the learned models are sufficiently accurate so in this case of gamegan we're going to need a really accurate world model in order to play pac-man inside of it in this research they're going to be predicting the future with the generative adversarial network framework rather than the popular variational autoencoder technique and this is because gan images tend to be sharper than the variational autoencoder images both of these models have support for random variable inference in order to do multimodal prediction so compare this with using say an lcm or recurrent neural network to predict the future they don't have a well-defined loss function for having multiple future predictions for the same input and this produces them to have blurry outputs because they're taking the l2 distance between all these possible multimodal outputs so the gan image uses the adversarial loss whereas this is an illustration of the variational autoencoder from an article on towards data science that shows how variational autoencoders deal with stochastic outputs and multimodal predictions what they do in the variational autoencoder is the encoder encodes the mean and variance parameters and this is sampled such that you have different latent variables for the same encoding quickly let's further describe why we use a generative adversarial network for multimodal predictions rather than a recurrent neural network a recurrent neural network like an lcm or an rnn would predict the future and then you would take the l2 distance between the ground truth image and then the generated image and the problem with this is that there are going to be multiple output images for the same input or it's going to be hard just to predict how much the future sequence will diverge such as in jan lacoon's example of predicting traffic there are so many different possible futures with respect to this driving sequence that if you take the l2 distance between the ground truth image you're going to get a blurry image so gans are really good at handling these multi-modal predictions by just having this loose adversarial loss that just classifies whether this is a horse or not or whether this is a zebra or not and this gives the generator the freedom to do this multimodal mapping so now we'll get into the gamegan architecture so the dynamics engine is an lstm that's trained in the generative adversarial network framework with three adversarial losses but overall the gamegan is a three-stage process we have the dynamics engine that takes in the action from the user the previous memory vector the image at the previous time step and then a randomly sampled noise vector and it's going to use this to update the cell state the hidden state and the lstm kind of way and it's going to send this hidden state to the memory bank the memory bank is inspired by neural turing machines and it's external memory that the engine can use to save the static background of the pac-man game so we don't want the generator to be changing the background of the pac-man game like rearranging the mazes or a certain like core components of the game frame you can imagine other games like viz doom we don't want the generator to be changing the background of the game so then the memory sends the memory vector and the hidden state from the dynamics engine goes into a third network the rendering engine which is going to use the spatially adaptive instance normalization layer that powered nvidia's gal gan model the one that's able to turn these pixel drawings into photorealistic landscapes first we'll get into the details of the dynamics engine which is an lstm to learn more about long short-term memory networks i highly recommend checking out chris ola's blog post understanding lcm networks so lstm networks take in the previous image of the current image in this case we're going to have a convolutional encoder map from the previous image of the pacman game into a low dimensional representation that goes into the lstm block and it's also going to take in the previous hidden state of the last state of the lstm because the recurrent neural network where you feed in the hidden state to the next step and so on to kind of auto regressively update this hidden state so we have the forget gate which is where we decide what we want to forget about the current cell state we have things like the input gate which is how much we want to gate this input from the x and the previous hidden state and we have the output gate so there are these different gates in the lstm which are ways of structuring the internal memory of the recurrent neural network architecture so for the dynamics engine in our gamegan we're going to be using this lstm architecture so first we construct this hidden state v sub t which is the hadamard product which is this element-wise multiplication or you can think of it like a masking operation between the previous hidden state and a multi-layer perceptron projection between the action vector the random variable and the previous memory vector then we're going to construct s sub t which is the convolutional neural network encoding of the previous image of the pacman game then we're going to update the internal variables of the lstm which the with the input gate the forget gate and the output gate and the cell state and then we're going to output this h sub t which is you know the product of passing these variables through these lstm operations so the second phase of our gamegan architecture is going to be to use an external memory module and the motivation behind this is that the model needs to remember every scene it generates in the hidden state we want to have consistency in the static elements of the game when the pacman ventures around the maze if it returns to the original location we want the maze to still be intact and it's not trivial to design a loss that's going to enforce this long-term consistency long-term consistency modeling is one of the hardest problems with these kind of networks with these lstms and rnns that's why we have these transformer models in natural language processing but it would be challenging to use a transformer for something like this because of the high dimensionality of these pixel frames you have 84 by 84 by 3 pac-man image frame so it'd be tough to use long-term modeling with the transformer so rather we're going to use this external memory module inspired by neural turing machines so to quickly recap a neural turing machine the neural turing machine receives the external input from the environment and then it has the controller agent that has this parametric read heads to the memory and parametric right heads to the memory so it accesses the memory with its parameters in order to update the controller's hidden state and then produce the external output so by using this memory it's able to explicitly store information without storing it in the parameters of the controller the memory module is going to use a learned attention kernel w which is based on the egocentric view of the agent or where the agent is within the kernel it's going to learn how to update these parameters as it's shifting and updating the memory in the memory bank the memory bank has m by m or the size of the game board and of these matrices that contain these features that w writes information into and then it has d of these m by m matrices so these matrices are just randomly initialized between zero and one by a normal distribution so something like this so the w kernel is learned throughout learning so you see in some cases when the pacman moves left it actually shifts the w kernel right and sometimes when it moves right it shifts the kernel w to the left so it's learning how to update its memory based on the view of the agent within its little kernel so in the memory module we update the parameters of our w kernel by using a softmax which is where we sum up the probabilities to equal one with this multi-layer perceptron k and then the action vector that the user who's interacting with the world model sends us then we have this g gate which is a function of the hidden state from the dynamics engine so what this is doing is supposed to mask out the kernel's update if we're taking a forbidden action like if we're trying to move left or right out of the borders of the game or we're trying to move down or up through a wall or something like that so then we're going to gate this with this function and then we're going to use the two-dimensional convolution with our new kernel and the previous alpha so alpha is what we're going to write to the memory also using this multi-layer perceptron encoding of the hidden state from the dynamics engine and then the previous memory bank for the arguments of writing to the memory and we're also going to get our new memory vector t by reading where we just wrote from the memory bank so just as a quick reminder of where we are in describing our gamegan system we've described the dynamics engine which is the lstm that takes these inputs and produces the hidden state output to the memory and the rendering engine the memory uses this learned kernel based on where the agent is and the hidden state from the dynamics engine to decide where to write in the memory bank m and then we're going to be sending this memory vector m sub t to the rendering engine the third phase in the system is the rendering engine which has the most complex architecture but all we're doing is we're dividing the two different contexts the memory vector and the hidden state vector from the dynamics engine splitting them and then combining them later on after they go through some processing so it's kind of heuristically motivated to take a transposed convolution and transform the vector into an object map and an attribute map the idea is that the attribute map would roughly describe the scene the object would describe particular objects then we also have a type entity encoding so it's all kind of heuristically motivated how to split up the different feature maps and recombine them so what we do is we pass the object map and the attribute map through the spatially adaptive instance normalization which is this way of combining the styles really well which is used when the nvidia researchers convert pixel maps into photorealistic landscapes so this is combined with the type encoding after it passes some processing as well and then it meets up again after it goes through these layers of transpose convolutions and these different you know operations and then they're summed up to get the next image produced from the rendering engine from the memory vector and the hidden state from the dynamics engine this video from nvidia showcases the power of the spatially adaptive instance normalization layer that's used in our game gan to combine intermediate features in the rendering engine in this case for converting pixel maps into photorealistic landscapes this spatially adaptive instance normalization layer is really good at combining the styles of one image with the semantic content of another the spatially adaptive instance normalization layer uses conditional batch normalization spatially so it's taking different scale and shift parameters along different x y locations of the c intermediate convolutional feature map that's produced by this one set of features and then this other set of features so in our case this is the combination of the object and the attribute map that's being combined with the entity type through this spatially adaptive instance normalization layer so in the generative adversarial network framework this is our generator we have the dynamics engine that passes information into the memory and the rendering engine the memory passing information to the rendering engine that produces the image to be passed to the discriminator is going to have these different loss functions on the generated image so all of this is differentiable we didn't have any kind of discrete operations that would prevent the gradient from flowing back from the generated image into the read and write parameters from the dynamics engine or into say the mapping of the encoding of a z and m in the original h function with updating the dynamics engine's hidden cell state so we're going to train this generator with three adversarial losses and a regularizing cycle loss on the memory so our adversarial losses start with the single image discriminator and this is taking one image x sub t plus one from the generator and telling if it's realistic so if it belongs to the set of pacman images or viz doom or the randomized maze pac-man environment and telling if this frame from the generator is realistic or not as in the standard general adversarial network framework the second loss is the action condition discriminator so in this case the discriminator receives these tuples of x the next x and then the action that was taken and this is done to make sure that the game engine is taking the action that the user took into account so if the user is moving pac-man left or right you want the generated image to reflect that action taken the way this is done is by having an adversarial loss between negatively sampled pairs so we have the true x next x action then we have the randomly sampled negative action so say the true action is left we would sample down and then if the discriminator can't tell this then we have a problem and then we'd also have two randomly sampled frames and then the action that was taken the third adversarial loss is the temporal discriminator this is how we make sure the sequences of frames has this kind of long-term modeling although this is really going to be enforced in the memory bank so what we do here is we have a three-dimensional convolution so three-dimensional convolution is a convolution that also looks at the time axis so we stack up these frames usually you would have a frame that's say 84 by 84 by three three being the rgb a three-dimensional convolution might be it usually is three by three by three where you would slide it through this frame and then it would go to overlapping the next like rgb axis so it has this time information and then you have the hierarchy of convolutions that can aggregate the entire time scale by these local time features from the three by three by three cnn so the fourth loss we have is the cycle loss so the cycle loss is encouraging the engine to put the static background elements in the memory vector by comparing the distance between what the rendering engine produces when it just takes in a corrupted memory vector and no hidden state compared to what it produces with the full input so now we'll get into the experiments in the paper we're testing this out on the pac-man environment with the ghosts and we're testing this out on pac-man with a random maze and no ghosts as well as the viz doom environment that was studied in david haw and jurgen schmidt huber's paper world models so some of the more technical details of the pac-man environment studied in gamegan is that it's a modified version of pac-man the pac-man agent is going to observe a 7x7 grid from the full 14x14 environment so it's not going to see all the ghosts at every time step the images from the 7x7 grid are 84 by 84 height width of this pixel map the actions available are left right up down stay and we're going to learn this world model from 40 000 episodes although we're sampling 45 000 and each of these trajectories in the pack in the real pacman environment are greater than 18 steps and this 18 stat all of greater than or equal to 18 steps in the 40 000 sequences of this is used to train the gamegen model so each episode consists of the sequence of the 7x7 pacman centered grids along with the corresponding actions that were taken in order to have that action condition discriminator these animations from han schmidt huber's paper world models shows where the prior state of the art is with these pixel space reconstructions of these games this is showing an illustration of the car racing environment from open ai in this interactive visualization from their paper you can play the car racing game in this imagined environment they also show the results on vis do so this is the true visual environment and now this is the vision environment that comes out of their variational autoencoder framework so this is the paper from han schmidt huber world models and this is what it comes up with for learning to map from this wisdom environment into the world model so you can play inside of this imagined game of vizdu this image shows a qualitative evaluation of the pacman environment with different kinds of models the first is an action lstm which is a recurrent neural network conditioned on the action taken so it takes the action as input and then models these future sequences and you can see it's starting to fall apart as the time sequences goes on you'd expect this to fall apart even further if you went out to say 30 frames this is the world model from hanschmidhuber which uses the variational autoencoder framework this is a reduced version of the gamegan with less complexity than this is the full gamegan model described previously as a part of the gamegen research the authors come up with two quantitative metrics to evaluate these world models the first of which is the obvious one to train and reinforce learning agent in the world model and then see how well it generalizes to the real world so this table shows the performance of these different agents trained in the world models you see the game game with the reduced architecture performs much better than the world model or the action lcm which basically is randomly guessing with the training inside of the virtual model and then evaluated on the real pac-man game but you see the world model in holland schmidt tuber's paper still outperforms a game game on the viz doom environment the second evaluation is to test the consistency by using this external memory module in the come back home task so similar to uber ai's research on go explore and then first return then explore we want our agents to be able to return to previous states in the simulation so this isn't quite like go explore but the idea is that we're going to try to return to a previous state that we visited and in this case we're testing our world model has the background become completely obliterated by venturing out and then returning to the state as you can imagine with the long-range dependency of modeling the sequence of frames and actions that the agent is taking in the generated environment so you see in this case that our gamegan model has the lowest variance with respect to this distance metric that's looking at the distance between the walls and this kind of static elements behind the pac-man game this table illustrates the qualitative evaluation of the come back home task so starting in this frame we venture out from the task and then return back to the original location or venture out from the original location and then return back you see the action lstm produces a slightly shifted wall compared to the original frame and then we when we've returned back to it in this case only the gamegan model is able to exactly reconstruct the original sequence by using that full external memory module whereas the game game with the reduce architecture has these slight corruptions as well as does the world model variational autoencoder another interesting characteristic of the gamegan research is the ability to disentangle the static elements like the background and the dynamic elements like the moving agents in the like the goes to the pacman agent or the disappearing food as the pac-man runs over it they show that you can disentangle the static background by replacing it with these other backgrounds with the disentangled architecture they further show that you can generate these different mazes in the second environment where there's no ghosts and you're generating random mazes by traversing the pac-man with the gamegan model so you can design these new mazes which may have applications for designing new games like some kind of this game doesn't exist as you scale up these models and imagine the next generation of this kind of research further this table shows the results of scaling up the capacity of the game gan and training it with more resources you see in this case it's doing a much better job of preserving the ghost compared to the previous visualization and you should check out the paper if you want to see more details about this described in the appendix a this is a really exciting paper that's looking at predicting the future in model-based reinforcement learning if we can construct these models we can learn in simulation inside of our own imagination and this obviously saves a ton of effort with respect to constructing these simulations that do things like robotic manipulation and then doing sim der reel from that so one of the key problems with these models is the compounding errors when predicting the future a great quote from this paper regularizing trajectory optimization with denoising auto encoders is that planning is effectively an adversarial attack against the agent's own forward model so as an agent is looking at its world model or its imagination and trying to plan out the best path it will quickly exploit these inaccuracies in the future modeling and then take the path that is sub-optimal because of the errors in the predicting the future estimation another interesting area of this research is how much knowledge can a neural network store in this paper researchers from google are investigating how much information for uh no context question answering they can put into the t5 text-to-text transfer transformer model in this case they're looking at answering questions just based on the weights in the neural network in our case of the game gan we're trying to store entire games in this generator architecture so it's interesting to think about how much knowledge can we store in these neural networks could we store something like the size of an xbox game into these neural networks it's all really interesting area for thinking about the capacity of these neural networks and what they can memorize with some sense of kind of interplaying between these different frames another interesting tangent to this research is gan compression if we want to be able to store pac-man into these neural networks we want to further be able to compress the generator such that we can quickly run the inference for the next frame in the pac-man we don't want to make the left decision and then wait for the freight the environment to render the next frame we want it to be super quick and we want it to have low memory requirements so i highly recommend checking out this paper again compression where they use things like quantization and knowledge distillation to compress the cycle gan down 21 times and the pix depicts model down about 12 times i've also made a video on the scan compression paper if you're interested thanks for watching this explanation of the gamegan neural game engine from researchers at nvidia in addition to the neural game engine and the excitement of being able to play pac-man of these different games inside of a learned world model i'm really excited about the potential application of this for model-based reinforcement learning hopefully from this video you got a sense of the generator architecture the dynamics engine the external memory module and then the rendering engine as well as the adversarial losses used to train this model some of the evaluation metrics they use as well as the overall concept of this kind of neural game engine and the idea of compressing these games that have all this if else logic and the physics equations into a generator neural network i think it's going to be really interesting to see how this progresses i also think it's going to be interesting to see how this stacks with different compression techniques for running these generators faster and with less memory requirements thanks for watching and please subscribe to henry ai labs for more deep learning and ai videos [Music]