ControlNET Stable Diffusion: Automatic Pose Transfer - Easy Guide

Key Takeaways

transfer any post to any of your AI images this method is called control net you can use it in automatic 1111 to use this pose and transfer to that image pretty amazing here's how that works so the first thing we need to do is to do some installing here and that is super easy on the GitHub page of control net you want to scroll down and here you have your install guide so the first thing we need is to install the CV2 library for that you want to copy this gray code here pip install opencv minus python then you want to go into your local automatic 1111 folder click up here in the address bar and write CMD this will open up your command window in that location now you paste that command in here pip install opencv minus Python and press enter this will quickly execute this install then it's done you can close the window after that what you want to do is to go back to the GitHub page of control net and copy the address of that project up here in your address bar then go to automatic 1111 click on extensions click on install oral enter the oral here and click on install after the installation is finished which only takes a couple of seconds you want to go here to installed and click apply and restart UI now you have installed control net but you still need the models now this is a bit of a downside of this method because these models are rather big so you can see here that each of these different models has a size of 5.71 gigabyte the good thing here is you don't need all of them just download what actually helps you create your art now when you download them of course you need to download locations so you go to your automatic 1111 folder click on extensions then you click on the folder as the web UI control net and in there on the folder models and here this is where these models go so you can download them directly into that folder now here you have that list I have linked everything below the video so you have here this Arrow that's pointing downward simply click on that and this will ask you where you want to install that file if you don't see a dialog you probably have set up your browser to download into the downloads folder so look there for that file and then just move it over into the file location I showed you before after that automatic 1111 is ready to go now the location where you can find this is simply by scrolling down and here you have a new tab it's called control net there's an arrow here on the side and here you have multiple choices the main choice for you is going to be to click on enable here so you can actually use that and there's also an option here for low we run so if you have a older graphics card or a graphics card with low vram you want to select that option too then you have here two choices you have the preprocessor and then you have the model now it is important that you select the preprocessor and the model of the same name you can try to mix them you get some interesting mostly Artful results that are not as intended so here you have several methods of pre-processing and then here you have all the models that you have downloaded next to this we also have a weight slider this is how much that technique is taken into consideration when you are rendering your image so the lower the weight the less of the posing is taken into effect and the higher the weight the more is taken into effect by the way if you don't see your models you have your big button it says refresh models you can click on that and this will update this list here then down here you have more choices for the canvas with the canvas height the envelope outer fit or inner fit or just resize in most cases I found that inner fit works pretty well you want to set this not to the same resolution but to the same ratio as your image it's also okay to approximate the ratio here so if you have a vertical image the canvas height is going to be a larger number of course and when you have a vertical image the canvas width is going to be a larger number now down here of course it says drop the image or click to upload so click on that so click on that and then you want to select an image you want to use as an input next you want to set up your prompt negative prompt and other settings here as usual and click on generate and you can see here that this has generated an image it has the same pose as our input image and it also created a second file that has an edge detection now what are these processes for the Kenny process is for Edge detection that creates finer lines around the most important edges that are found in your image this can be very important for high detail Precision but also is very good when you want to have for example anime cartoon images drawings things like that the depth process is for a depth map this gives you a form of spatial awareness in the image but it has less details in there so if you want to recreate this spatial positioning of characters and objects in your AI scene without also taking the details this process might be helpful for you the head process creates also Edge detection but you can see here it doesn't use Fine Lines it creates more of a blurring outline around the most important edges in your image now this is often in the result very similar to the Kenny process but I think it works better with photography rather than with drawings because as you can see here with the map that this is creating it can pick up on finer details because it has a gradient in there for example like hair the mlsd process is for straight lines which you mostly find in objects and architecture so if you want to recreate the building this is the process to go for the normal process is creating a normal map that you might know from your 3D projects from 3D textures gaming textures things like that now this is more about the height inside of the image and this can be very useful again if you want to create different dimensions different volumes in your image without all of the smaller details that you can find in the image the open pose models are for figuring out the pose of the character now this is mostly for when the person is spread out and as the model says you have an open pose but what this also requires that there is no overlapping and no hiding of the extremities the arms and the legs of that person because otherwise the process cannot identify that position so in most cases you want to have a person or character that has a outspread pose and has tight clothing where you can see all arms and legs very easily the scribble process is for actual scribbles that you can do with a pen or digitally and this can convert a rough scribble into an anime image into a painting into a realistic photo the fake scribble on the other hand will take a photo input create a fake scribble from that and then apply it as the scribble method the benefit in this case with the fake scribble is that the finer details are removed because the photo is converted into a scribble and then used as a an input there's also the segmentation method where you actually need to install on top of that the pretty table I don't know actually what that is for if you know let me know in the comments now let's get back into automatic 11 11 and look at some more things we can do here first of all the question is where do you get all these images with the poses now of course you can start by a photo page like unsplash where you can find a lot of images of any kind of activity of sports of dancing of portraits and you can use this as an input for your posing another interesting method here is to use the free software days 3D I will link that below the video there's a bit of a learning curve here on how to understand that but you can post 3D models in here fairly easily in any kind of position you want export the image and then use it as an input one of the added benefits here is as you can see here in the background that this will create a gray figure that is also unspecific in its features so that might be easier to apply to different images that you want to create of course another thing you can do is to go to deviantArt because there you will find a rich pool of thousands of photos that are meant for drawing now the benefit for those is that you already have very interesting poses and that the model specifically wears skin tight clothes that is also neutral and because of that these poses can be much easier picked up by control net and another added benefit here is that these poses are specifically for artistic purposes so they have often poses that you will find in concept art for games but also for classic paintings another great resource if you want to invest a little bit of money is reference dot pictures here you can buy large bundles with thousands of images in there with classic poses but also action poses and the good thing here is that they also have some free sample packs so for example with this pack you get 45 high resolution images with all kinds of different poses in there last but not least that is important to also point out that this method can be used in image to image so here in the image to image tab you would as usual have your Source image and then when you scroll down you have your control net tab down here and you can have your pose image with the settings as before now in this case the way this is controlled is both by the image to image denoise strength the higher the value is the closer this is to the pose but also it is more different to your input image why when you have a low denoise value this is picking up less of the pose but looks more like your input image and in addition you can play down here with the weight of the control net method that I have found personally that this method often works very well if you have an input image then you have a sketch and you mainly want to transfer the color scheme of your input image to a new pose of course in that case the phase of the person is going to be different than in your original image unless of course you're using a model you have trained with dream Booth or you're using a Lora model and last but not least you can also use this with a dream Booth model you have trained or with a Lora model to put your own face into the AI art with the position you have chosen for that to work what you want to do is to describe the person in the image in my case bold man with a beard then here the command for the Laura model wearing a white suit then down here in the control net tab you want to load the pose you want to use and in this case you want to set the weight rather low because otherwise the face of the person or the characteristics of the person is overwriting your image so in this case I'm choosing 0.3 as my weight and you can see the result is pretty amazing and it is still very close to the initial pose let me know in the comments what you think check out my Facebook group and thanks for watching bye oh you're still here so uh This is the End screen there's other stuff you can watch like this or that's really cool and yeah I hope I see you soon uh leave a like if you haven't yet and well um yeah