Get FULL creative control over Stable Diffusion | Install + all models

Key Takeaways

there is an extension that by itself makes a stable diffusion better than mid-journ and today we're gonna install it it is controlling but if you already knew about it you may want to stay because there's new stuff and I'll also cover all the models that there are and what to use them for given that some of you may already have this to update control net go to extensions check for updates and after a few seconds take the ones that you want to update and apply and restart the UI once it is done you should have control net version 1.1.173. if you didn't have control net it is as simple as copying this text that I'll leave in the description going to the extension install from URL and pasting it up here then just click install once it's done go to installed click check for updates and again apply and restart UI and now you have control net but you still need the models to go along with it and as you can see there are a lot of them but don't worry I'm gonna teach you what they all do so you can decide by yourself which ones you need you can download the ones that you like the most by clicking this and then move them to the extensions control net and models folder I will separate the models by groups based on what they do there will be the add detection group The mapping group and the modifier group I will also go over open pose which is a must if you want to have people in your images to show you how everything works I've modeled this quick 3D scenario I'm going to move the render into the image space inside control net if you want text to image to match the inputted images Dimensions click this Arrow icon right here and it will adjust your height and width accordingly if you don't have an initial image you can also click this and take a picture with your webcam or create a canvas and paint something inside of it this works well with the scribble model that we will see later on the way this will work is this you will have an input image then you will choose a preprocessor and a model each preprocessor will look at the input image and extract certain information from it for example The Edge detection will try to extract the lines your image creates in high contrast points and then this new extracted information will be fed into the model every model fits off of different types of information but preprocessors usually match the name of the model you use them with you can also input an image with the information already extracted and use no preprocessor at all as there is no need for it in that case you can see why this is a powerful tool now this extension allows you for infinite control over what you want in the image and how you want it to look like controlnet plus some other basic skills like photobashing or drawing is almost an invincibility hack there is no way a yard setback that can hold you back now next you have these options clicking enable will activate control now check low B ROM if your PC isn't Omega good I have it active anyway as I haven't seen much of a difference in quality either you use it or not if your resolution doesn't match the input resolution you can try activating Pixel Perfect this will try to match it automatically without you having to worry and finally allow preview I recommend having it active to see what your preprocessor is getting from the inputted image for example let's start using the first Edge detection model can now we can click on this explosion icon and it will generate a preview as you can see this is what we are telling stable diffusion we want the image to look like and it will try to follow these edges adjusting the control weight we can give it some Freedom at one it will follow the edges as they are and the lower you go the more it will be able to deviate from them having it at more than one will put a ton of contrast where the edges are not really recommended you can also adjust the weight by adjusting the starting control step or the ending control step if you have played with prompt editing at some point you may find this easier to understand if not go check this video the starting control step is at what step you want control net to start affecting the result I think it is a percentage of the total steps so 0.1 would be 10 this is what the image without control net looks like and this is what it looks like when control net starts at 25 of its steps then this is what the image looks like if control net decides the composition and then fully disappears at 25 of its steps this is a part of the Kani model which I won't go into this video because each model has its own special parameters to play around with and finally you have the control mode I usually keep it that balanced but if you feel like control net is taking too much out of your prompt then click the my prompt is more important option and if your prompt is blasting through the control net click control net is more important resize works like an image to image so just what this video if you need more info now we have seen what Kenny does and it is really good to have control over how many details you want to maintain from the original image on the same line okay I'm sorry if you just want to maintain the shapes but not necessarily the details you can use soft Edge which is like a cunny but more diffuse you can also turn your original image into a line art with the line art or anime line art model which treat the input as a drawing to be painted I'd use one model or the other based on what style of image you want to create really boxy shapes like interior designs or some isometric builds you can use mlsd this will extract the known curved lines and feed them to the model so basically straight lines to just catch the overall composition you can use fake scribble and match it with the scribble model this model allows you to input a super simple drawing and it will interpret a new image from it big scribble basically creates that quick drawing but from an existing image next you have the mapping models these models try to extract information of how things interact with each other inside the image for example you have the normal map you can see that it is creating some weird colors like green purple red Etc and if you don't know what's going on here that's why I created this 3D scene here I can show you the real normal map of this image separating my colors green is providing the information of the top of the objects the parts of the image that are facing upwards and then the other colors Define a different axis so if green was the positive y then these ones are positive Z or positive X this is pretty good if you need cohesiveness on how the lights affect the planes of the objects or to maintain the 3D shape of your input another Super useful model for this is that this one tries to capture the distance between objects relative to the camera I would advise activating the preview and play with the preprocessors that there are until one of them gives you a good result keep in mind that white means close to the camera and black means far away these models try to guess this thought obviously you can see here that this big sphere is closer to the camera than the main tower when in reality it is further away as an example I'll use my own depth map that I rendered having in mind that it needs no pre-processing but Maya renders depth in Reverse so black is closer to the camera in this case I'll have to use the invert preprocessor to have the depth map applied correctly now you can see that everything is in place and keeps its overall shape the last Model is segmentation or SEC it acts like an ID model where it detects each object and assigns it a color value based on what it thinks it is you can actually see what each color value means in this page right here I really have to thank Olivia sarikas for this uh really sorry if I pronounced their name wrong sorry as I didn't know that this existed until he made a video on it after this you can check that video out if you want more info this allows for really precise changes on each part of the image with just a simple prompt super good if you have simple easy to recognize objects that you need to change specific things about them I actually planned talking about the color and the style model but they don't work for me anymore so they basically took the images colors or style and then applied it to yours you can't find this on the main page I gave you so I'm gonna give you the other page where they are just in case they work for you because they don't for me now we go into the modifier their job is to create a variation of the image input for example Shuffle will take the colors of your image distort them and use that as as a base to create similar images text to pix is a pretty fun model here you can take the original image and mask stable diffusion to change something about it for example here I typed make it snowing plus a scribble model with the image and that changed the result so it had a snow all over the place reference only is a new preprocessor that acts as a model at the same time this comes installed with a new update and it's super good to create images that are really close to the original it is able to maintain key aspects of the input really really well so try it out if you want to use two models at the same time you can go into settings control net and increase the multi-controllenet slider to the desired amount then just restart the UI there's also a tile in this group but it is a really good upscaler so I'm gonna use it later on when we have a good image to upscale and to create that image we will use open pose this model makes it so you can post your characters any way you want including the face and the hands even though it isn't super precise with those yet and there is 5 pre-processors for this open pose alone will take an image with a person on it and try to create a skeleton rig that matches their pose without hand or face just the head and body here I use this image with open pose and SAC at the same time you also have open pose hands which will create the same skeleton but also looking at the finger's position and trying to replicate it like here if you were wondering where do I get this poses here's a little trick as thanks for staying until this part of the video you can go to mixamo and look around for animations in 3D you can pause these animations whenever you want and take the frame that you like the most while also controlling the camera angle and the distance then just take a screenshot drag it in and that's it next there is the phase model you can have it with the full body or just with the face alone open pose 4 will get everything in the image face hand and body okay now let's just upscale the image for this I'll send this new picture into image to image I'll also add it in control net and use the ultimate SD scaler as the script download link in the description below then I'll choose the tile preprocessor and model what this will do is divide the image in tiles and upscale each of them individually to get more details in and then it will mix every tile together I'll choose the upscaler that we downloaded in the last video put control net in control net is more important and then just adjust the settings like this not sure if putting pixel putting to the max helps but why not I need to activate SIM fix though this should make the tiles seems less visible don't be afraid to experiment that I'm not 100 sure on how to use this at its best and now you have the best extension for stable diffusion we still aren't done with having full control over the image we generate though so make sure to watch this video if you want more Precision if I have to say preprocessor once again I'm leaving YouTube hey if you're still watching Please Subscribe see ya