Stable Diffusion ControlNet Tutorial

Key Takeaways

hello and welcome back yet again in this chapter we're going to be digging into my favorite part about stable fusion and generative Ai and that is control net because in stable Fusion compared to any other tools like mid journey and Dolly you have total control of your image and there are many tools to achieve this we're going to look at one in particular today and that is the extension control L and with it you're going to be able to take an image like this and replicate the post and just create images like this and this and this it comes pre-installed on think diffusion so let's get to [Music] work we are back inside think diffusion and here we have a loaded session of automatic 111 and we haven't delved much into all the tabs that are in here um there's just so much to cover with inst stable diffusion and think diffusion you have all these different extensions that get you total control of your image and today we're going to look at one down here which is named control net and if you click that you're going to be able to expand that and here you have a option to drop an image and I prepared an image which is generated so here we have a ballerina dancing and it's going to be a pretty complex image because of the hands here they aren't going to be perfect but that's not the point here don't mind the hands we're going to be replicating her Pole post so let's first just drag this image into the little box here first off we're going to be using something called open post we're going to select open post here and we're going to make sure that our model here correlates to the model that we're using up here now we're not using an Excel model here which it can be seen as it's not excel in the name that means it's a 1.5 model so then we should use a 1.5 model here the pre-processor is open post and we can see what happens if you press the I icon here so this is a pre-processor working and the output here will be the open po so it reads from our input image the ballerina here and outputs this skeleton now we're going to enable the control n here if your image is different from the size that you're going to Output you need to select Pixel Perfect as a rule of thumb you can always use Pixel Perfect it's not really going to mess with your image at all the control weight here is defaulted one and that's how much control net will influence your image you can also select when it starts and when it stops so if you have here for example now we have I'm going to change this to 30 sampling steps so if you set this for example to starting control step .5 that will mean it 50% of our render open post will start to work so at 15 Steps control that will jump in here and then run to the end this it's an any control step if we set this to starting to zero it will start from frame zero and run the Whole 30 steps we can even set the ending control step to.5 that will end at 15 we're going to run it for the whole full 30 steps here for the control mode and the resize mod mode here we're just going to leave everything default if you feel that you aren't getting the results that you're looking for even though you have a high weight and you have full control steps here starting and ending you can select control net is more important but most of the time it's not necessary so we have the pose here what are we going to do with it well let's start off with a prompt here and let say cowboy I'm going to select 1.5 Styles so I'm going to load a digital painting and default negatives we're using a width and height of 512 x 512 I'm going to select the highr fix here which is basically an image to image pass and you can select how much bigger your image is going to be and this is upscale by two so R 512 x 512 is going to be 1024x 1024 we're just going to leave all this default for now but you can change the steps here to fine tune your results most of the time it's going to be the D noising strength you can check one of the previous chapters to learn more about D noising strength we're going to create four Images here now let's get these five images running as you can see here coming in from the preview we are getting a similar looking post but uh let's get these images finished and uh catch up with you in a bit so we have our four Images finished now here we have our first calar and you can see this the same pulse here here here we have another one you can see keeping the same pulse again this one keeping the same PSE here we have some Modern LED lights coming in and this one I mean it looks fairly okay the hand up here is a little messed up but uh this is really the power of control net and how you can keep a pose consistent now this is only using the open post so this can create any type of image so we could change this into green monster boy and let's change something instead of the digital painting let's try in the game here for example and let's quickly just generate this again so here we have our results we didn't really get our monster we got this green boy here using the same pose in a more of a indie game style here I think it looks uh looks pretty cool now again this is just using the open post let let's try something different so let's say for example you have an input image where you want to keep the visuals and not just the post so you could for example load the cany here and if you click the little pre-processor you can see the lines so if you look at this image for example you have this jacket and the new hair and everything if you use a canny you would stay consistent with what you had here so let's try this we're going to change it into futuristic sci-fi astronaut we are generating again here now for images and as you you can see as it's coming in live here now the outlines of the woman's body is staying consistent with our new astronaut so this might or might not be good for you it's going to be much harder to actually create like an astronaut's outfit or suit so it's going to be very slim you're not be able to have those big large NASA space suits that you can see in the movies but if you're trying to go for a specific look this can be a great way of using control nuts and as you can see in our outputs here now the waist here and and the legs and everything is staying consistent to the original now there are more control net options available one very popular here is the scribble or the sketch let me quickly find an input for that now here you can use anything that you have drawn even though it's a very crude sketch like the penguin that I've drawn here maybe it also looks like a man inside of a refrigerator but you can probably draw much better than I can this is also a great way taking a child's drawings and making them into great looking art and if you again press the like in here you can see the output now we lost some lines down here so we could change the scribble to X dog for example which will get us a little better looking result and let's change the prompt here into penguin again make sure that everything is enabled here and we are going to generate four new images and as you can see coming in here live our penguin here is matching very well with our crudely drawn sketch now there are ways to getting penguins that are getting a little bit more realistic compared to your sketch here and that would be to changing the values down here so I think we're going to do that for our next step but first let's see our images once they are completed so here we have our penguins and they are consistent with the sketch that we did so that's very cool let's try a different set of prompts here let's select the original photo style and let's actually change here the ending control step so we're going to leave that to 50% here so control net is going to run for 15 Steps and then it's going to be able to run without the control net so the point of this is giving control net a base to work with and then for the last 15 Steps stable Fusion is going to have more control to go a little bit beyond your image and that will give you well a better looking result in the end most likely especially if you have drawn as crudely as I have if you're a better Sketcher well you can probably run it for the 100% duration and look looking at the results here they're looking a little bit better especially this one I would say looks very good and it's not now as constricted with the body and the the arms here or the wings that we had in our original drawing now let's remove this let's head back to our ballerina input here and let's let's look at the depth map here for example now if you again press the icon you can see a depth map coming in of the image and this would be similar to the canny however it's still a little different so you can mix and match and if cany doesn't work for you you could try the depth map for example as that will give you a different result you can also use multiple control Nets so if you go into the second tab here enable this one you can drag the image in here again you can select uh open po here for example so this will give us both the open POS and depth map so I recommend you going into your control net doing stable diffusion inside the think diffusion try what works best for you I've shown you some of the most popular control Nets and how to use them and I think you're going to have a great time with control net as it's really really powerful that'll be the end of this chapter and I hope you've learned something that you can use while creating your amazing images