VAE-GAN Explained!

Key Takeaways

[Music] he's watching Henry AI labs this video will explain the variational auto encoder again known as the VA he can framework the high-level motivation is that auto encoder has taken an input image and then encode them with neural networks into a low dimensional vector representation they then have a different neuron that were called the decoder that takes the vector representation and reach has reconstructed into the original image the way that not Oda coder is trained is based on the pixel wise distance between the reconstructed image and the original image the idea behind the variational auto encoder in the gaen is to get towards a more semantic loss function than the pixel wise distance metrics so ganz used the generator and the discriminator framework such that they can have a more semantic loss function that avoids the element-wise similarity measure another interesting loss function for the reconstruction is to use features from preacher and classifier so for example if you have a ResNet 50 that you might train on image net classification you could look at the features on say layer 48 and the difference between the original image and the reconstructed image and you could use this to evaluate how well the decoder has reconstructed the image from the low dimensional vector representation so the overview of the variation auto encoder gann is like this you take an input image and you encode it into a low dimensional vector and then this little mention of vector is going to be regularized with the reaper a memorization of the variational auto encoder and the KL by Perkins from a normal distribution will be explained in more detail in the next slide and then you from this low dimensional vector you decode it with the generator you never place the decoder with the generator they become the same thing in this framework so the generator takes the low dimensional vector and it produces an image from it so now the evaluation of this image is done by the intermediate features of the discriminator as well as the discriminators task of is this reconstructed image or is this one of the original images to review the differences between a regular auto encoder and a variational auto encoder first start with the audio audio encoder an auto encoder takes an image in and goes into a low dimensional vector and then decodes it into an image a variational auto encoder modifies this framework by mapping points to a Gaussian space rather than like some direct point in the low dimension vector space so the encoder has two outputs it outputs a vector of means for each dimension so let's say you have a five dimensional vector that you're trying to map the image to you would have mean one mean two mean three mean forming five and then in addition it forever eise's the covariance matrix of the Gaussian but in the variational auto encoder you assume that there is like no correlation and then it's a diagonal covariance matrix so really you're just outputting a set of variances so it'd be variance one variance to various three and this is how you remember eyes the encoder so that it just structures the space of how it encodes the data into the low dimensional vector so it's structured better and it has much nicer properties for interpolation which is the idea of using a variational auto encoder to as a generative model so that you can design new images new audio from the low dimensional encoding space the gender of adversarial network framework is another approach to generative modeling unlike the adversarial auto encode I'm at the variational auto encoder it doesn't encode the images into a latent space rather it just samples random vectors from a normal distribution this is an interesting similarity with the variational auto encoder because the variational auto encoder uses the KL divergence with the parameterization of the meanin variance to sort of enforce the encoder to align the data in a normal distribution so in a way the Z vector sampled in the generator and the Gann are pretty similar to the vectors encoded by the encoder and the variational auto encoder so the Gann adds this discriminator versus generator component to the training which is something that we would want with our variational auto encoder the discriminator looks at the generated images and the original images and just classifies which one is which and then they optimize each other in the adversarial game so this is the overview of the variational auto encoder plus the gam together you start with the input image and then you encode it into a low dimensional vector the way that these low dimensional vectors are encoded is based on the reaper a memorization of the variational auto coder and they're trained with the KL divergence from standard normal distribution so the generator network then samples this Z vector and uses it to produce a reconstructed image so now the way that the reconstructed image is compared with the original image is through the intermediate convolutional features and discriminator as well as the discriminators classification of real first fake putting this all together there's three loss functions used to train the model there is the loss from the prior this is the loss used to enforce the encoder part of the variational auto encoder to be near a standard normal distribution this is the KL divergence between the encoded latent space and then a normal distribution the second loss term is the difference between the reconstructed image and the original image in terms of their the intermediate features on the discriminator and then the final turn loss term is the generator says it's real or fake so this is the flow of losses during training and so somewhat interestingly the discriminator doesn't use the intermediate loss function in its loss optimization because this would just cause it to look for really subtle details and then also an interesting note that they put is that they find that when they use the gam loss and the encoder they get worse results compared to not using it and I thought that was kind of interesting I didn't really understand why they found this result so to train the decoder they have this interesting analogy with the neuro style transfer how they weight the content of the image and the style so they reason that the intermediate feature distance between the reconstructed image and the original image is like the content of the image like how similar is it to the original image and then the gann loss is like the style of the data set really because it's not quite particular to this one individual instance so putting it all together the algorithm works as false you have parameters for the encoder the decoder and discriminator you sample a random batch of images you encode the you encode the batch into the low dimensional vector space and then you have a loss function on how the vector space was constructed then you to code the images from the vector space then you have this loss function that measures how this dissimilar the reconstructed images are from the original images in terms of the intermediate features on the scrim inator and then you have the Gann loss function and then you have the updates to the parameters with this waiting on the decoders update and then also interestingly you see which lost terms go to which models this is the convolutional neural network architectures used in their VA EGH and paper in the encoder side they use a series of convolutional layers to take an input image and compress it into a vector of dimensionality 2048 the decoder flattens takes this vector pass it through a dense layer and then uses a series of up sampling convolutional layers to arrive at the final image and then the discriminator also has a similar architecture to any other convolutional neural network most notably it's it's not very deep it's only like five convolutional layers and two fully connected layers so the data set that they test their VA egan model out with is the celeb a data set this is a really popular data set for testing out generative models mainly because they're all centered in the middle to frame there's not too much variance and it is set and also it's pretty large data set it's got about two hundred three thousand images they each have forty binary attributes attributes describing like characteristics of the face such as eyeglasses like the hair and then so they're all 64 by 64 resolution which isn't too difficult for generative models so these are some of the results from the standard variational auto encoder the variational auto encoder that adds the intermediate discriminator feature loss to the reconstruction metric the combination of the variational autom could RM began and then a generative adversarial network by itself so more interestingly is with variational auto-encoders you can have a training and a test set like you can train the z embedding space on the encoder part with the training set and then you would pass in a new image from the test set and see how well it reconstructs it so it's pretty interesting to see how the in this case you do see a pretty significant improvement over with the via egan compared to the standard variational auto encoder and then another interesting result is the way that they can interpolate in the late in space like they in their paper they claim that this is a disentangled representation they've learned but I guess I'd say the modern view of disentangle representations means that you just have you know one component of the vector that you can shift and change it the you know the attribute but would they find the said as they would take they would use the binary attribute labels to find the mean vectors of the images in the latent space and then they would take the average of this so it's like they're adding the bald vector to an image but this vector has values in all the dimensions it isn't just like a disentangle you know one single row of the vector so as a future work that they talk about is replacing the second loss term with the siamese network although it would require label data and then they talk about instead of using the discriminators intermediate features what if you use like ResNet trained on image net classification or something like that thanks for watching this explanation of the BA Egan please subscribe to Henry AI allows for more deep learning videos