Installing ControlNet when using the Stable Diffusion Web UI (Automatic1111)

Key Takeaways

welcome to foffer AI the channel where we explore the world of artificial intelligence and its latest tools and techniques in this video I'm going to show you how to install the controller extension for stable diffusion when you're running the automatic 1111 UI I'm going to assume that you've already got that running and you don't need instructions on how to do that personally I'm using stable diffusion through a Google collab so that I can generate my images a bit faster to install the contron extension you need to go to the extensions tab and you need to go to available and you need to load all of the available extensions from this URL once loaded you get a list of all of the extensions that are available and here you will find the controller extension and you'll click install so I've already installed the extension it's called SD web UI control net so you'd find that in this list and then you click install just as a note once you've got it installed you can click check for updates this is a an extension that's being updated pretty frequently so I recommend doing this occasionally just to double check that you've got the latest version newer versions add new features for example being able to use multiple control Nets at once then if your extensions are out of date it will show that they're behind you can click apply and restart UI to update each of those extensions this is the GitHub repository for the control net extension that we're using it has some instructions on how to install so open the extensions installed from URL this is what we did once you've downloaded the extension you need to get the controller models and put them into the right directory before you can run them so you can download these pre-trained models from hugging face so these are the original control net models each of them is about 5.7 gigabytes and you can see each of the different model types these are large files and they're also pickle files so I would recommend downloading the smaller extracted versions of these models which are about 700 megabytes and the safe tensor versions so they are safe to use so this is where you can download the safe tensor versions of the control net modules each of these is 700 megabytes um so I've downloaded from this one through to this one now you need to make sure you save these models in the right directory when I started I put the models into the stable diffusion folder which is where I've been putting most of my models but that's not the right place for controlling it to find the right directory you need to go to the extensions directory that is part of automatic 1111 within that you should already find your sdwebui control net folder which will have been put there when you installed the extension and inside of that there's a models directory that's where you need to put all of your control net models that you've downloaded from hugging face now if we go back to our stable diffusion web UI and our extension is installed from the text to image tab there will now be a control net section at the bottom so by default you just have a single control net when you install it so you can open this section and this is where you control control net the first thing to do if you want to start using controller is to enable this module so you need to click enable if you don't do this then control net will not be guiding the diffusion process then we need to pick our preprocessor so our preprocessor is what takes the image that we've put in this section and process it into the correct format for the model to interpret so let's say kany which is a form of AD detection and then we can choose the model that goes with it so Ctrl K you can mix and match these so you could take something that's pre-processed with scribble and put it through the canny model for interesting and strange results so let's pick an image for our contronic I've chosen this mid-journey generation of a prince and princess [Music] let's try a prompt where we turn them into a cyberpunk couple [Music] I'll say 50 steps I'm going to go to my favorite sampling method and our width that we're going to go for is 912 for a 16 9 aspect ratio we've got our control net it's enabled we're going to use canny Edge detection we're not going to change any of these settings and we don't need to use any of those buttons so controller is there it's enabled we've got our prompt we've got our sampling we've got our width and height set correctly we've got our sampling method set correctly let's generate our image and this is our image output we can first look at the pre-processed image so this is the result of taking this image and putting it through the preprocessor with the Canon model and we can see all of the edges that is detected this is then what guides the diffusion process with the prompt so our prompt was painting of a cyberpunk couple and this is the result you can see it's very similar to the original and we've still got those sort of pink circles and the crown and the pose so you can see how using control net you can really maintain aspects of your original image if you want to know more about what each of these pre-processors and models does check out my previous video in which I go into quite a lot of depth about each of them and what they're good for and doing some comparisons of image Generations using those models one more thing to show you um when you install controller you start with just the one uh control it if you want to add multiples if you go to settings and you find the settings for controller and there's this setting here multi-controller Max models amount and I've whacked this all the way up to 10 but you can set it anywhere between 1 and 10. and that's what allows you to then run multiple control notes alongside each other and you can pass in the same image or different images you can use the same models different models the more models you use the slower it's going to take to generate your image but the more interesting the results thanks for watching hopefully you found that really useful and you can now use control net if you've enjoyed this video please like And subscribe thanks very much