Stable Diffusion ControlNet Explained | Control Net Examples

Key Takeaways

hey friends welcome to one little coder you have been hearing about stable diffusion with control net almost everywhere everywhere you go if you are a stable diffusion fan everybody has been using control net so in this video I'm going to try my level best to explain you about control net and show you things that people have been doing with control net it is quite amazing what people have been doing with control net and we are going to see every one of them and to start with first let's try to understand what is even control net so if you see control net it's a very simple thing I mean it's not simple so control net is a neural net architecture so it's a new neural net structure that helps you control diffusion models like stable diffusion model by adding extra conditions so to show you what control net can do I have come up with a very weird example but uh if you have not watched the movie Logan it's a spoiler for you so if you have watched the movie Logan what happens that they have taken Wolverine's DNA and then made some modifications and came up with this girl so that's what the entire Logan movie is about so you have got Wolverine who is actually a natural mutant and they have taken the natural mutant and then they have they have made this girl from Wolverine's DNA and this is exactly what control net can do so control knit can take an existing stable diffusion or diffusion model architecture and then just make slight changes to the architecture and then add whatever you want to just like what they've done with this girl here so if you now look at control net examples so you can upload an image and then ask control net to hold some properties of the image and then change other properties I'll give you the easiest example that everybody has been looking at so for example if you look at this this is control and stable diffusion control net with the pose so you can see here that you can upload an image and it is going to preserve the pose it's not going to do anything else it's not going to preserve the scenery it's going to preserve the pose and it is going to come up with new process or New Image just for the same pose based on the prompt that you give and it is not only for post like you would have seen a lot of examples of posts lot of people have been showing examples of pose but it works for a lot of other things and that's like you can see for example uh here you have got a and it can hold the scribble map the fake scribble map and from that it can design new things and you can see a lot of examples here how it can hold it just how it can you know take a simple scribble and then build images on top of it so you can see a lot of examples here but I'm not going to show you the examples here but at first I want to show you the growth of control net like if you have been wondering why everybody is talking about contournet you know right now that control net is quite amazing in holding certain properties of a neural network and even to say how it does it actually makes a copy of a neural network just like in Logan it makes a copy of a neural network and it holds one neural network and it makes changes in another neural network and when you get the final output it has both the properties that's why when you upload a picture of a man standing you can hold the pose and then instead of a man you can put a woman you can put a guide you can put a kit you can put a robot with the same person you can do anything so that is happening because now there is a duplicate copy and then it combines further so now this is about controlling now if you simply look at control Nets growth this has been tweeted by hugging face CEO you can see that control net like what Plymouth said is that there are already 50 public and open control net models on the hugging phase model Hub it has got more than 100 and 1200 likes and you can see the growth of control net is almost as exponential as stable diffusion so people have been really grow going crazy with what control net can do and this is a great example of what control net can do and you can see how the trajectories growth rate is and you I will link in the YouTube description the models and the demos and all the links that we are seeing so you don't have to take any notes so you can just go to the YouTube description just below the like button and check everything that you want and if you also see uh this is the control net model is available on hugging phase modeler but if you don't want to use it from hugging face model then you can go to the control net GitHub repository and then start using it it's quite amazing what it can do it has a lot of things people are still figuring out every day what they are doing so now that we have learned about control net what is control net or control net explanation now that we have also learned about how control net is growing now let's go and then look at some examples of what people are doing with control net so first you can see just search for control net you can see these kind of animations where people have taken an image with the different poses and then they are trying to use control net and also sometimes with blender epicent lot of different applications and try to combine those images like for example you can see how you can take an image take the pose and then create certain aspects around it and then now you can create another image so this is one thing that people are doing the other thing that people are doing is control net with Nerf Nerf control net with Nerf so this is by beloved so you can see this tweet how you if you have to emulate a drone shot people would use typically drones you know robotic arm cameras like this but here this is all simply screenshots or photo snapshots and control net and bunch of other things the workflow has been shared I will link it in the YouTube description and I think this is this is another new trend that a lot of people have been doing use control net image and ask it to you know capture either the the edges or so what you can do is you can upload a logo a brand logo and ask control need to capture the edges like we have seen like there are certain control unit models that can keep the edges intact and use that to now come up with new Landscapes new images this is this is quite amazing from a brand perspective let's say you want to make an advertisement copy and you want to show your brand logo in the middle of a dessert in a tennis court a football code somewhere and this is quite amazing like you can see how they've naturally embedded Nike logos in like literally any place that you say like a hill station desert a volcano a landscape see wherever you want and this is all not very difficult to do this quite easy to do and all it requires is the right control unit model that you want to use with stable definition and the right prompt that you want to give and and uh again this is another popular use case a lot of people have been playing with this I've made a shot about it it's called a scribbledefusion.com you can go to scribbledefusion.com you can scribble something and give a nice prompt and then it is going to create that prompt as a result for example now if you think this looks like I mean you're you think that you're actually making a cat so make a cat like this and then you know mention whatever you want like cat wearing a call by hat and it is going to help you create a cat wearing a copper hat because we have actually seen that control knit can create that or hold that scribble map and then from that it can it can generate new images so like I said the applications are amazing so the latest one is how you can use control net to create a movie or a scene like for example the problem that people have been having with stable diffusion is control stable efficient is really good but creating consistent scene like having a control over what you want has not been very easy with stable division always so that's where control net comes and plays a very vital role imagine like you want a character to be on like this and now how do you do it it's quite simple with control net you need to just create the post put the post and then that will do it and you can see certain examples like how you can feel like you're a movie director or you can make an animation using controller and put the characters in the right place and finally I would like to show you another important um Discovery by dushyant actually it's not just you can give an input image and extract the pose and then create an output image you can also create your own pose and then based on that pose you can generate image like for example this is a pose that was not extracted from an image but rather it is a post that has been created with the right colors for the open post model and using that you can actually create any image that you want and I mean like I said I can just keep on going on so people have used control with dream booth for example if you have got your own model or if you have got a model for let's say celebrity or um and your advertisement company I've got a dream Booth model for somebody now you can use control net and place them in such a way that they have been posing for that thing so overall control net is quite amazing it is taking stable division completely forward into a totally new place so if you want to use control net the easiest place to start with is the hugging face models and hugging face demos I'll link it in the YouTube description there are control net extensions for automatic 11 11 UI that is also something that you can explore I mean it to make Hands-On tutorials with controller but I thought I need to actually explain what is controller doing here and how what kind of potential control net has if you have not started using control net I think today is the best way to start with it is quite amazing I hope this video was helpful to you in learning probably a Logan spoiler but also learning about control net and what control net is actually doing here so there are a lot of examples about what kind of properties controlling it can hold so I'll link everything in the YouTube description and I'll also link all the examples that we discussed in the YouTube description please check it out try out control net if you have any question let me know in the comment section otherwise I hope this video was helpful to you see you in another video peace