Generative Modeling Applications

Key Takeaways

what if you could design an avatar what would be their story maybe you could generate a theme song for them or create a 3d model of them to explore a virtual world this is the power of generative modeling if we can learn a probability function we can use it to generate images music and even 3d objects hello world it's Suraj and machine learning in 2019 has gotten really good at generating data some of these techniques are truly mind-blowing and Vidya for example was able to generate an entire 3d map for a video game automatically in this episode we'll learn the basics of generative modeling then use that knowledge to build three apps that each generate music images and 3d objects respectively and anyone can get these up and running since the code lives in the cloud via Google collab you don't even need to install anything on your local machine links to each demo will be in the video description there are a lot of different ways that we can categorize the machine learning process does it have a label does it not have a label does it apply to simulated environments does it blend these are great frames they help us understand the techniques we should use given the type of data we're dealing with but a more application centric frame is does it generate or does it discriminate to paint a better picture here let's say we have a labelled data set of integers X is the input Y is the output a discriminative model would want to learn the probability of Y given X in probability theory we define this as a conditional relationship since one variable depends on another independence is overrated AF so if we iterate through each data point and check if given that x equals one what's the chance that y equals zero it's 100 percent for the first data point and 100 percent for the second so the total is a hundred percent now how about when we're given x equals two if we iterate through those data points where that's a given we'll find that only half of them have y equals zero thus the probability is one-half now a generative model is one that will instead learn the joint probability distribution of a data set the probability of x and y and this will output different values in the previous probability distribution if we take the case of x equals one and y equals zero we'll find that these both occur together twice out of the whole data sets thus the joint probability is 50% in general given a data set discriminative models learn the boundary between classes whereas generative models model the distribution of individual classes across the entire data sets they accept no boundaries boss status a generative model models both the features and the class if we model the probability of x and y we can use this distribution to generate data points all algorithms that can model the probability of x and y are generative and we'll talk about a few of those examples in a second if we had a data set of song audio files and their associated genre labels we could build a discriminative model to learn the conditional distribution of the data so that in the future given a song we could predict its genre a generative model more interestingly could given the genre generate a related song there are three main types of generative models these days auto-encoders adversarial networks and sequence models auto-encoders attempt to generate an output that's the same as the input they compress the input into a lower dimensional representation called the hidden space it's called hidden because it's compressed data not because of some ancient secret then they attempt to reconstruct the output from this representation we use two neural networks here an encoder network that learns the probability of the hidden space given the input and a decoder model which learns the probability of the input given the hidden space which will reconstruct the input as the output adversarial networks involve a generator model that learns the probability of x given the learned hidden space H where X is the input and a discriminative model which learns the probability of Y given X which tries to associate an input instance X to a yes/no binary answer Y about whether the model generated was a genuine sample from the data set we were training on or not the counterfeiter versus the detective and both improve over time the counterfeiter is what will help us generate the data set we need how bittersweet and let's not forget sequence models these models learn the probability of the form y given a specific location in the sequence and a given input sample as an example we can consider each word as a series of characters each sentence as a series of words and each paragraph has a series of sentences the output Y could be the sentiment of the sentence using a similar trick from autoencoders we can replace Y with the next item in this series of the sequence namely y equals x plus 1 allowing the model to learn let's now go through how these models work in practice first I found this really cool Google collab for generating piano songs collab if you don't know is a way to run Python code in your browser you can easily train models on a cloud GPU without having to install anything python beginners rejoice in this collab we can play with a pre trained generative model to make it do several things we can generate an entire piano performance we just need to select which key we'd like it to be in and given some initial notes we can generate the rest ourselves we can even generate an accompaniment for a given song you can imagine how we could download this pre-trained model then use it inside of a web or mobile app to serve people we could create a music making tool for artists or a music collaboration social network or we could just generate our own unique theme song without needing to hire a professional the team at Google made this possible by using a type of sequence model that's very popular lately called the transformer model their whole process was pretty clever starting with how they generated the data set to learn from they started with a collection of YouTube videos that all had a license allowing for their use then they used a model to classify those videos that only contained piano music nothing else their classifier was trained on audio set an audio data set that contained over 600 event classes and a collection of over 2 million 10-second sound clips drawn from YouTube videos hopefully not ASMR videos though super useful data set by the way they use its piano data to learn the sound of piano music in general no matter the order of the notes and collected hundreds of thousands of piano music videos as a result they then converted those audio waves into symbolic mid AI format which represents musical notes digitally within 10 K hours of symbolic piano music there is more than enough for a machine to learn how to play the piano the transformer network has an encoder and a decoder it's a sequence model input a sequence of notes and it outputs a sequence of generated notes in terms of what that collection of specific matrix operations in both the encoder and decoder do that's way too much to fit into this video but they are just math operations different combinations have been tested by AI researchers before coming to this conclusive ordering because this is the one that gave them the highest accuracy scores they built the model using magenta using its various operations to build these matrix operations step-by-step linear algebra makes life worth living magenta is a collection of musical tools built on top of tensorflow it's got pre-trained models that make it really easy to generate music it's learning the probability distribution of a music data set over time this is the learning process now if we wanted to instead of generating music generate an image say of different types of faces how are we gonna learn that probability model will need to give a machine a set of facial features and see if it can generate a unique face in that style I found a collab that lets you do just that no need to type in any code at all give the image a starting face and have it transform into another over time or give it some initial starting features and have it generate faces it's better than drugs it's machine learning it's using a program or progressive growing generative adversarial Network and video released one of the coolest papers ever when they announced it it takes the idea of again but introduces a unique training technique for it they progressively grow the size of both the discriminator and generator networks they are mirror images of each other and grow in synchrony new layers are added to the networks over time while it's training on an image datasets they train it on the celeb a image data set a collection of celebrity facial images as the resolution increases in the code you can see that it adds more convolutional layers in a loop nested inside the function use to build each Network that's a pretty clever trick convolutional layers are matrix operations that tend to perform well with image data thanks young this reduces the training time it's two to six times faster depending on the final output resolution we could imagine that we could use this kind of technology to create a talking head video that can give a weekly educational video lecture on YouTube generating mouth movements frame by frame wait maybe I should just automate my own job now I love doing this too much clearly transformer networks are pretty good at learning a probability distribution for musical data and programs are pretty good at learning a probability distribution for images but what if we wanted to generate 3d objects well I found a really cool collab that allows for style transfer for 3d objects basically you give the network a 3d object and a related image and it'll generate a new 3d object that's in the shape of the input in this style of the related image what a cute little paper demon bunny how it does this is by using a convolutional net work to extract both the style and the content into separate representations the style represents the textures and colors whereas the content is the 3d coordinate collection and through the optimization process at every step an output 3d model is rendered from a random angle from two different locations the original one and the learned one the content loss optimizes the approximate positions of the pixels whereas the style loss approximates the visual patterns without regard to the positions together through gradient descent it achieves a generated object this probability model we learned the probability of x and y where X is the 3d object and Y is the style image outputs some gorgeous visualizations and we can play with inside of the collab I think I'm in love the way that they implemented this is by using a really pretty library called lucid by the creators of this still it allows for some cool transformations with lucid you can visualize all the learned features inside of a neural network and implement various style transfer applications one of the best documented github repositories I've ever seen definitely check it out as for 3d datasets we could use object net 3d by Stanford University which is pretty good we could generate new types of characters for 3d video games as a service for developers or small companies we could also generate all sorts of fashion accessories giving the customer the utmost personalization for their own specific style or have it automate interior design work I've added the link to all three of these web demos in the video description for you to play with and find some inspiration from there are three things to remember from this episode you can generate data if you learn the joint probability distribution of a data set generative models are capable of doing this for lots of datasets out there and some examples of generative models are adversarial networks transformers and convolutional networks [Music]