Style Transfer Better Than GANs! Swapping Autoencoder Explained

Key Takeaways

this new technique can change the texture of any picture while staying realistic using a complete unsupervised training the results look even better than what gens can achieve while being way faster it could even be used to create deep fakes let's see how they did that and the amazing results [Music] this is what's ai and i share artificial intelligence news every week if you are new to the channel and want to stay up to date please consider subscribing to not miss any further news researchers at berkeley university recently introduced a new technique for existing images manipulation in the recent paper called swapping auto encoder for deep image manipulation where they propose the swapping auto encoder in short it's a deep model designed specifically for image manipulation degenerative models such as gans are the state of the art in term of image manipulation however they require the task to be defined a priori and need extensive training data which are both unconvenient when it comes to modifying an existing image plus these gun based methods learned a mapping from an easy to sample typically a gaussian distribution to the image domain enabling the generation of random images in the target domain this is the reason why this method was created their main goal was to learn a model designed specifically for image manipulation rather than random sampling while allowing both global and local image editing the main advantage of this technique is that it is fully unsupervised requiring no human supervision like class labeling or object localization needed with gans methods to achieve this they have to train what they call to be a swapping auto autoencoder shown in this picture consisting of an encoder e and a generator g with three main objectives first it needs to be able to reconstruct the image accurately then it needs to learn independent components that could mix together to create a new hybrid image finally it needs to be able to unravel the texture from a structure by using a discriminator that learns co-occurrence statistic of image patches let's detail this a bit more the encoder forms a mapping between the image and the latin code using the encoder e and the generator g while this is done the latent space created by the encoder e is divided into two components that are intended to encode structure and texture information during training the structure code learns to correspond to the layout or structure of a scene while the texture codes capture properties about the scene's overall appearance this is a huge difference with the recent gan models it is a huge advancement in computation time for such tasks gans attempt to make this latin space gaussian in order to enable random sampling while their idea was to use swapping constraint that will focus on making these distributions around a specific input and its plausible variations instead of following a gaussian random distribution the third segment the co-accurate patch statistics is then applied in order to create a result for image editing where the structure and texture will be both represented correctly by the components of the model this is done by using a patch co-occurrence discriminator that enforces the output and reference patches to look indistinguishable this is all done by using both resnet's architecture and a style gun 2 design both of these are linked in the description for more information now let's see what this technique can achieve and the comparison with the other state-of-the-art methods as you can see this method reconstructs the images much faster than the generative models can with much more realistic and impressive results just take a minute to look at these amazing results [Music] [Music] this technique is even generalizable it can change the texture on faces better than guns of course this was just a simple overview of this new paper i strongly recommend to read the paper linked in the description for more information please leave a like if you went this far in the video and since there are over 90 of you guys watching that are not subscribed yet consider subscribing to the channel to not miss any further news clearly explained if you want to support the channel i now have a patreon linked in the description where you can do that thank you for watching [Music] you