Understand Stable Diffusion Interpretation using Cross-Attention by DAAM Heatmaps

Key Takeaways

hey what's up coders welcome to one little coder in this table diffusion tutorial we're going to learn how to understand how to interpret a stable diffusion image creation if you are not familiar with stable diffusion stable diffusion is an AI system that helps you create images generated R generate pictures generate thumbnails whatever sort of images that you want you can generate it using stable efficient stable division is basically your open source version of Dali imagine like that so now what you do not know is how stable diffusion is making a particular image generation for example if you have got a prompt that says an angry bald man um you know reading a research or something do you know how does it understand angry how does it create angriness on the picture or which part of the picture is referred to on greenness for stable Division and this exact package or this tutorial that we are going to see today is going to help you understand that particular aspect of what is a particular piece in your image that stable diffusion linked with the particular prompt that you gave for image generation this paper is called what the dam interpreting stable diffusion using cross attention I'm not going to get into the technical details of how this is happening but we're going to see how to implement this for your own stable efficient model first I've got a Google collab notebook before we move forward please start the repository and thank the developer who is in this case question thank you so much for making this amazing Library I think this is purely gold for us to understand what is happening inside stable diffuse this library is going to work only with python 3.10 so for that you need python 3.10 so if you see my Google collab notebook the first two cells or a hack for us to get Python 3.10 and that is thanks to a stack Overflow user who has made it possible for us so thank you thank you coraco for making this amazing notebook which we can use for 5.3.1 10 on Google collab so how are we going to use it the first thing that you have to do is when you open the Google collab notebook which I will link in the YouTube description below the like button so you can click the notebook make a copy of the notebook click file make a copy save a copy in the drive and then go to runtime and then change runtime and make sure that you've got GPU selected and make sure you have got python 3.10 selected once those things are available then run the first cell and before you move forward with the second cell restart the runtime restart the runtime and then run the second cell that should ideally show version 3.10.6 if that is done now your Google collab uses python 3.10 the next step is for you to install Dam the library that is required for us to do stable diffusion in interpretation once dam is installed the next thing is make sure that you've got diffusers Transformers grade you also dependency scipy installed most likely all this should have been already taken care of and if not just reinstall it again after you do that the next step is for you to import required libraries from Dam we need Trace set seed plot overlay heat map and expand image from diffusers we need the standard import stable diffusion pipeline from matplotlib we need pipelot to display the plot and then import torch at this point we have imported all the required libraries the next step is for us to define the stable diffusion model for which we want the interpretation in my particular case I'm using the mid Journey V4 diffusion model that is available on hugging phase model Hub one of the reason why I decided to use this model for this particular tutorial is because this does not require authentication so if you're somebody new who has not seen previous videos of stable diffusion then you can still use this notebook without any authentication but if you want to use either runway model or if you want to use the compass model make sure that you are going to authenticate your notebook with hugging face token before you come to this particular step if you have any questions about this please let me know in the comment section I'll try to help my best the next step is to set up the device which is in this case Cuda because we started with a GPU in the first place the next thing is stable diffusion pipeline from pre-trained this particular model and then get the model and then move the model to the device in this case which is Cuda the next step is for you to define a prompt in my case I've said an angry bald man reading a paper with a pair of glasses so I'm expecting a picture where an angry bald man is reading a paper and the man also has got a pair of glasses next for reproducibility we're going to set the seed value once we set that first thing is we're going to generate the image as you can see we're going to generate the image using the same pipe which is quite common if you have been using diffusers Library the next thing is we're going to split our token so our token is here an angry bald man reading a paper with a pair of glasses so what happens when you split your prompt is prompt dot split you're going to get a list of individual tokens we're going to iterate through those individual tokens so that we can create a heat map for every single token and overlay that heat map on the existing image that we got created from stable diffusion so what are we going to do for token in prompt split which means but reiterate through every single token we're going to say you want to generate the Heat app for the for the entire image and we're going to use the function called plot underscore overlay underscore heat underscore map to take the input image which or sorry take the image that has been generated using stable division use expand image using the token that we have got and then add the token as the title and then show the plot this is quite simple if you are interested in knowing the details of the source code of what does this function containing you can go inside this library and you can read about it but if you are just like me more interested in understanding how stable diffusion is learning something then you can run this code which would ideally work and at this point you should have a lot of images for example for all these tokens that you have got it is going to create an image so and so it's saying and so you can see that and doesn't make any sense for stabilization the next thing is angry you can see that it is used the face to show anger next is bald you can see that it is showing the forehead and the head and the next thing it shows man and you can see I think it's probably pointing to the mustache and then reading you can see now it's pointing to closer in between the paper and the man and the next you can see a again doesn't make any sense then paper paper is being pointed out as paper with again it's something in the middle and then a doesn't make sense a pair doesn't make any sense of not required glasses and you've got glasses and that's it so this is a very simple way for us to understand how stable diffusion interprets The Prompt that we give or how to link or how to map the stable division image that is generated and the prompt that we gave one important thing that I learned from the paper that they have published would strongly encourage you to read the paper here is that stable diffusion is quite bad in handling quantity or numbers and that is something that they've proven in the research and you can see the examples in their paper as well and that is something that we have seen multiple times especially when you tell stable efficient to create something like five units of it doesn't do it properly and that's that's that's quite easy for us to understand from this but you know having said that let's create a new prompt and then understand something so I'm going to say uh I a very happy little boy working on a computer and I'm going to say run this set the seed split the prompt for us to just see a very happy little boy working on a computer and then I'm going to run this it's going to take a couple of seconds because it's going to first generate the image and then based on the image it's going to create the worldly and for every token it's sorry it's going to create the heat map and for every token it's going to Overlay that particular heat map value so a again doesn't make a lot of sense happy you can see that the happiness is associated with the face I don't know why the head is coming here that's okay happy a little maybe the fingers are little okay that's fine and a boy okay now it's identifying this part with the boy not no idea working um I ideally would have expected it to point out the fingers but it's not on a computer so the computer the Entire Computer is being marked here which is quite good and also surprisingly you can see the fingers have been handled really good in this so what I'm going to do now is I'm going to go to lexicon.art and I'm going to pick a prompt which is um let's say this Emma Watson so I'm going to copy this prompt and then I'm going to come back here and I'm going to paste it here so I want to paste The Prompt here and I'm going to run this run this run this and this it's going to again take some time but we can see how each of the prompt value or prompt token prompt word is going to help us understand how a particular image is being mapped as the image gets generated you're going to see first of all Emma I'm quite surprised to see that just simply showing writing Emma creates Emma Watson which is which is quite surprising um this is Emma maybe when you add Watson it is getting more specific as Hermione um so again nothing much nothing much nothing much nothing much nothing much one final thing which is to say in the prompt I'm going to finally say cats and dogs playing football [Music] let's see what it does whether it identifies cats and dogs and football specifically this is a very simple prompt again our objective here is not to create the best image but to understand or map the stable efficient images okay this is a cat okay it's cool cats and uh and okay dogs playing football and you can see the football cat and the dog is probably somewhere behind on the football and this is exactly how you can learn what stable division is doing and also you can optimize your prompts and also you can like for example if you're going to start a service to use stable division now you know how to trim the token for example if you know that a cat or you know anything like that doesn't make a lot of difference then you can start trying to trimp the prompt and then see how the generation is but overall this is an amazing research tool especially for somebody who wants to understand what stable efficient is doing this tool damn this is a really good tool but the dam interpreting stable efficient using cross attention I would strongly recommend you to at least skim through the paper it's in London it's it's it's it's really nice it's a nicely written paper make sure that you read it otherwise there is a hugging face demo that the author has put together and this Google collab notebook will directly help you use this library on your Google collab for creating your own stable efficient images and then see how a particular word is mapped to a particular component or part of the image that can help you improve your prompt sustainable efficient creation if you have any questions let me know in the comment section otherwise give a thumbs up subscribe to the channel share it with your friends and make sure that you start the repository and give a shout out to the developer see in the next video take care happy coding