Intuitive Intro to Image Generation - Stable Diffusion Masterclass

Key Takeaways

welcome back to the Staples or course on diffusion models I'm Mandy and in this lesson we'll get an intuitive intro to image generation with latent diffusion models we now have a general idea about what latent diffusion models are and the various components involved with generating an image before we get into all of the technical details behind what's actually happening to generate these images we're first going to go through an intuitive introduction and to do that we are first going to consider an image classifier as we know an image classifier accepts images as its input and as output the classifier will assign probabilities to various classes for which it's been trained on to specify where it thinks that the given image best belongs among these classes so for this example here let's suppose that we are working with an image data set of grayscale images of gym equipment and our classifier here is just a binary classifier so it's going to let us know whether or not the given image is gym equipment so if we have this classifier that's been trained to classify images as being gym equipment or not then we can actually make use of this model to generate images of gym equipment so how on Earth might we do this the upcoming explanation for this is inspired by a lecture given by Jeremy Howard at fastai he's provided a very intuitive motivation for understanding what's happening whenever a latent diffusion model generates an image and the link to his video is in the corresponding lecture notes for this lesson if you'd like to check it out further so let's continue with this example where we have a trained image classifier to classify images as being gym equipment or not and suppose that instead of passing an actual image of gym equipment that we pass this very noisy 28 by 28 image to our classifier and as a result since this does not look like gym equipment and the model has been trained to classify images of gym equipment it tells us that there's only a five percent probability that it's it that it is assigning to this image actually being of gym equipment so a very low probability that this is an image of gym equipment and so what we can do is we could iterate over each of the 780 4 pixels in this 28 by 28 image and make each of these pixels either a little bit darker or a little bit lighter and then pass that updated image with the little bit darker and lighter pixels to the model and see how it changes the probability that the model assigns to that image being gym equipment and we'd be doing this iteration of darkening and lightening the pixels with the objective to get the model to actually classify an image as a piece of gym equipment so if we look at T equals 0 here we are at the very beginning this is our original noisy image and here we have skip to time step 150 and this is what our corresponding image looks like at this point so we can see that we are just illustrating that we've darkened this section of pixels here and that we've increased the model's probability that it assigns to the image being gym equipment to 2 12 percent from the original five percent and we can continue this pattern so here we've skipped another 200 steps we're at time set 350 where we've lightened these pixels in the middle and darkened the pixels on the top and bottom to get a 28 probability and we can continue doing that until ultimately what we've arrived at is that we've carved out what actually looks to be an image of gym equipment in this case this dumbbell here for which at that point the model is assigning a 99 probability that this is an image of gym equipment so essentially we are darkening and lightening the pixels in an iterative fashion and each time we do that we're passing the updated image to the model and seeing if this lowers or increases the probability that the model assigns to the image being gym equipment then we take that feedback and we either reverse the lightning or darkening or increase it if it if it improved the probability that it is that the model assigned to the image being gym equipment so by observing the output probability that the model assigns to the image after each iteration on each pixel we can find which direction to incrementally move each pixel so that the model continues to increase the probability that it assigns to the image being a piece of gym equipment so what we're actually doing is very increment mentally refining the noisy image until it becomes what actually appears to be gym equipment so this seems like it would work intuitively right but it seems very tedious to do so rather than us manually doing this image refinement let's consider having a neural network learn in which direction and by how much to move the pixels this concept is actually very similar to updating the weights in a neural network until we get a desired output but in this case we're having a network learn how to update the data of which our pixel values in this example until we reach our desired output all right so now we have this goal to create and train a neural network that will learn which pixels to update and in which direction in order to make a supplied noisy image actually look like a piece of gym equipment for this example so per usual we first need to obtain a training set and for this example our training set will need to consist of images of gym equipment and then we'll go through the process of adding various amounts of noise to these images and the amount of noise added to each of these images will vary from being just a little bit of noise to being so much noise that we can't even see the hidden image underneath and we've Illustrated that here where we have several different images here of gym equipment and here on the left we have just a little bit of noise where we can make out what the underlying images here kettlebell in this case this is image seven and then image two we have a little bit more noise image nine we have a lot of noise and image 4 here we have so much noise that we can't even tell what the original image is underneath all the noise so our training set consists of images of gym equipment that have varied amounts of noise applied to them so the next step after we have our training data and we've processed it by adding this noise is going to be to actually pass the input to the model so as input we are passing in our noisy image and as output the model is going to actually give us what it predicts as being the noise present in the input image so we're simply passing noisy images and then we're asking the model to predict the noise that is present in those images so then we do the typical neural network training that we're used to where we calculate the loss between the predicted noise that the model gave as output and the actual noise that is present in the input image and then of course we calculate the gradient of the loss with respect to all of the weights in the network and then we update the weights with those gradients and we continue to iteratively repeat this process allowing the network to get better and better at per predicting the noise present in the training data now once the network has been trained to accurately predict the noise in the training data then we can just subtract the predicted noise from the original input image to get the clear input image that is hidden underneath all of the noise and we can see that depicted here with this example so we have our noisy training image that we were just using above and once the model becomes so good that it can accurately give us the noise that is in this training image then we can subtract this noise from the training image to get the clear training image with no noise now once we have this trained Network that can accurately predict noise then we can use it for inference and during inference we can pass it an image purely of Noise with no image hidden underneath of gym equipment so why would we do that well let's see what that might actually look like this network has been at trained to identify noise present in images of gym equipment and so during inference it will output what it thinks is noise in the random noise image that we're passing here which are going to leave behind pixels in this noisy image that ultimately will appear as a piece of gym equipment so we're passing in this image full of random noise and because the model's been trained to identify the noise present in images of gym equipment that's what it knows how to do as the predicted noise in this image that's just full of noise it will essentially carve out a piece of gym equipment from the noise and identify everything else as being the actual noise present in the underlying image and so then once we subtract the predicted noise from the model which is depicted with all of this gray here from the random noise in the input image then we will get a new generated image and this new generated image is going to be made up of all of the data in the image that the model didn't identify as noise and is essentially going to look like a piece of gym equipment in practice the type of neural network that we use to do this process is called uned and recall we were introduced to unet in an earlier lesson and this is the main component of staple diffusion responsible for actually generating images so we got briefly introduced to unet earlier now we're getting this intuitive idea for how it's working to generate images we're going to continue expanding much more on that and understand the technical details in a later lesson so now we have an intuitive idea for how diffusion models are trained to generate images if we train the model on noisy images of gym equipment for example then the end goal is that the model will ultimately be able to accept an image of random noise and generate an image of gym equipment from that noise in practice however the data sets of images that popular diffusion models have been trained on consist of all different types of categories so not just gym equipment for example or related images to gym equipment but all across the board so how do we guide the model into generating an image that we actually want rather than just a random image from the number of categories for which it's been trained well we do that by in addition to passing a noisy image to the model we also pass in text input specifying what we want it to generate this text Will guide the model in it to generating an image of what we actually want rather than just a random image from the number of categories that it's been trained on and we touched on in an earlier lesson how this text input is processed before being passed to unet using the clip model as a text encoder and the clip will encode our original text input of what we want the model to generate in to an embedding and we also know that in practice the images that we are passing to the model are compressed into latents before being passed so here we're depicting just this image of random noise along with text well before it's being passed through the model this noisy image is being compressed into a latent using the encoder portion of an autoencoder and the text is being encoded into embeddings using the clip model and then as output the model is giving us this output image but actually as we've briefly introduced before this output is a compressed version of an image as well so it is going to be a latent that needs to be decoded back into an image and to do that we'll use the decoder portion of the auto encoder so we're being introduced to slowly all of the components and what's actually happening under the hood to generate an image we're getting little pieces of information bit by bit and the technical details will soon begin unfolding to understand how exactly all of these things are working together to give us a generated image but hopefully we have an intuitive idea of what's going on for how we can ultimately pass an image full of random noise along with a text prompt of what we want that noise to be turned into and how a model can essentially carve out a generated image from that noise