GANs explained | Generative Adversarial Networks video with showcase!

Key Takeaways

start hey there nice of you to stop by the ai coffee break in today's video we will explain how generative neural networks or short guns work no not guns gans but first let's see a little compilation of five examples of what gans can do well they can generate new data and they can do it pretty well for example the stylegan 2 can generate high resolution images of faces of persons that do not exist looking only at the face this image looks stunning but with a closer look at the background we see some weird things going on like what is happening around this man's neck is that a hand scarf are we even allowed to show this on youtube the artifacts become unnoticeable in cases where the background is just blurry amazing but also kind of creepy knowing that this person does not exist let's see does this website give us also pictures of women oh yes there she is are we courageous enough to see another one oh no please not a brain background oh quickly let's go to some other examples of guns doing crazy stuff like the pix2pix scan that can generate real-world images from edges that you can draw yourself just visit a website to generate your dream creepy cat yourself or check out this other pix2pix application that generates building facades based on coarse drawings where windows columns or other elements should be placed not bad but it can also go the other way around where from natural images freeze g a style gan 2 based gan generates some kind of doodles a la south korean cartoonist lee malnun the failure cases look funny it's art freeze g can even translate images to imagined simpsons characters and of course miss coffee bean insisted to show also the failure cases they are well also this gang can translate human faces to dogs wow this is what i have always wished for also perhaps you have seen this face the pixelizer that can generate the most likely high resolution image of a face given a data set and because this method can hardly be better than the data it has been trained on the phase d pixelizer is notorious for its bias examples the most infamous one is where it generates a caucasian male given a pixelated picture of barack obama speaking of bias glands can do harm unintendedly miss coffee bean now could give a last example of using gans for deep fakes but instead she wants to show her favorite usage of gans on video data the restoration or even better the perfecting of very old footage take this awesome example of 4k resolution at 60 frames per second of 1906 san francisco achieved with a commercial software called gigapixel of course we do not know exactly how it works but we have a strong feeling it is also based on gans for even more awesome videos check out dennis shiraev's awesome channel now we finally come to this video's topic how do guns work there are a lot of gun types and you have seen some in action style gun picks to picks and so on and so on but what do all have in common the original gun model architecture it was introduced first in 2014 in the paper of young good fellow and collaborators linked in the description below wow 2014 that was ages ago a gun is based on two neural networks working against each other does the name adversarial what's cool about a gan architecture is that it can generate data it's in the name but also do so in a self-supervised learning setting this means that the data is not annotated and that the model creates its own annotations and learns from them this is what the discriminator one of the neural networks does now we will see how the only thing we need is raw data like bare text if we want to do natural language processing or just images if we want to do computer vision these samples we call real because these images come from real data but for gants we also need some fake data that is coming from somewhere else mysterious for now let's bring the discriminator onto the stage this is a neural network with an architecture of your choice it can be just fully connected layers with the sigmoid for binary classification at the end this discriminator has to predict whether the image that is being passed to it as input is real or if it is fake so if it is coming from the other set of data that we did not explain exactly where it comes from well this data comes from the second neural network called the generator player 2 has joined the game the generator is also a neural network of your choice because however it looks like it has to generate these fake samples but where does it generate these fake samples from well from scratch this is in many cases just some noise variables in this so-called latent space that the generator produces samples from but how if we remember how neural networks work we know how much is in their parameters the generator takes one number from this latent space multiplies it with a parameter sums it up with another number multiplied by another parameter passes it through a nonlinear activation function and repeats this for all layers and to put it almost anecdotally let's say we want to generate a picture of a cat the generator learns the right parameters to multiply the latent space with in order to generate grass pixels for the background the grass pixels are nothing but rgb values so for the green channel we want a value as close as possible to 255 and we want zeros in the red and blue channels this would be the ideal case but of course at the beginning it would generate only noise from noise so how does the generator learn to produce grass or cats it has to take feedback from somewhere on how well it does right here is where the game starts between the discriminator and the generator the discriminator's job is apparently easy not knowing from which image pool the input is coming from it has to predict if the image is coming from the real data or from the generator if it succeeds to tell pictures apart the discriminator has one if the discriminator predicts an image coming from the generators being real then it has lost and the generator has won this is why this is a non-cooperative game the two are playing depending on the errors the discriminator makes the parameters of the discriminator are updated in order to perform better next time at the beginning one would guess that this task is easy because the generator's garbage is not hard to tell apart from real pictures but every time that the discriminator gives a verdict for a picture of the generator to be real or fake the generator uses this feedback in back propagation to update its weights and perform even better next time and this alternating training of the generator and the discriminator in their game continues until the generator is producing images of such quality that the discriminator has no other choice than randomly guess if the images are fake or real or at least this is the ideal case because in practice gans are notoriously difficult to train to convergence it's not impossible but very hard it only works if the discriminator and the generator learn together and grow incompetence at the same rate if one becomes far too good for the other at any time then it is like playing chess against your grandpa he always wins by high margin and you have no idea how or why one of the most common problems is called mode collapse this happens when the generator maps the latent space to the same output producing high quality outputs with very low diversity another thing that makes the convergence harder is that the nash equilibrium and with it convergence is not guaranteed what in english please we remember this is a non-cooperative game that a discriminator and the generator are playing since they want to fool each other the win of one side is the loss of the other the nash equilibrium is a game theoretical term named after the mathematician john forbes nash jr for describing a stable state of the interaction of the participants in which no participant can make gains by changing only their own strategy and this is exactly what is not guaranteed in this non-cooperative game it suffices if only the generator changes strategy and starts to produce suddenly perfect images because then the discriminator has no more chance to recover it will think that the stunningly generated images are real and all feedback it gets will be negative conversely if the discriminator becomes very good the generator will only get negative feedback with no chance of finding back the right track and to combat these and also other problems the whole gan zoo was established guns have been amazingly applied in computer vision what about other fields like natural language processing well there some more tricks are required for training gans because unlike image data that takes values from 0 to 255 text data is not continuous and neural networks don't like that and difficult workarounds have to be found so while guns are also successfully used in nlp there will be no amazing examples like the ones in the beginning for nlp and guns it was just not meant to be since all nlp attention is going towards the transformer attention pun intended but hope is not lost since the next amazing gun application in nlp could be done by you let us know in the comments if you would like to see a video with the math behind gans or let us know about any other topics you would like to have explained simply ok bye [Music]