ControlNET to Video - Stable Diffusion Automatic 1111 Tutorial

Key Takeaways

how to create videos with control net by the way you should absolutely sign up to the Nvidia GTC online conference the sign up is for free but they giving away a 3080 TI to my community so check out the link that I've provided below and as you can see here they have some pretty amazing speakers like the co-founder of Deep Mind the co-founder of open AI so there's a lot of amazing things to learn check it out and maybe win the graphics card hello my friends how are you doing big shout out to Timmy he posted this on Twitter and I wrote him and said hey how did you do that so he showed me the method and now I'm going to show you and this is working in automatic 1111 with control net check out my video on how to install it and how control net works also you want to check out this video where I show you how to use control net with Laura now let's check out the video method here and this is not working with the Forum as you might think instead this is working with image to image with the batch image processing but before we get started here of course you need a video so how do you get that and how do you split this into J packs well a very good source for videos is pixels you can search for everything here here we have search results for dancer you want to click here on videos over here in filters you can also decide if you want to have the orientation horizontal vertical or Square and you can also choose the different sizes you want to have after you found a video you like to use a very easy method to split it up into pngs or jpex is is easy.com here you have the video to GI tab you want to click on that and then on the far right you have video to jpack or video to PNG click on this you can see you can upload videos up to 100 megabytes after uploading the video you want to scroll down a little bit here it says the start time and the end time so you don't have to use the full video if you don't want to and then here for the size this goes up to a resolution of 1,200 pixels on the long side now this is very helpful for us because for automatic 1111 and control net the resolution of the input images can't be too high because otherwise your RAM is going to fill up and it's going to crash below that you have the frame rate personally I would suggest to you to go with 10 frames per second not with 20 frames unless you want to wait all of the time I use 10 frames per second and then in my video editing software I double each frame so I get up to 20 frames per second after you've done this the next step is to click here on the convert to jpeg button here the red button says download frames as sip you want to do that and then unpack that zip after that you have a folder like this now in this folder you don't want to have anything but these images and then you can also put the other folders of your test runs but no other files so now let's talk about how to set all of that up inside of automatic 1111 of course like I said watch my video on how to install control control net and then you want to go to settings and in settings you need to go to control net down here on the left you click on that and then you select here do not append detect map to Output now what this does is usually every time you use control net it's creating an image and a detect me next to it now if these are appended to it they cannot be replaced so you would only have one detect map so by choosing this the detect map gets thrown away with each image when you've done this click here on apply settings then you want to go to stable diffusion and check with image to image to exact the amount of steps the slider specifies click on that so you get a blue check mark click on apply settings and next you want to go to user interface look for the quick settings list and in here you want to use this text that I'm showing you right now on the screen it's also in the video description click on apply settings again and go back to image to image now up here you have two new choices one is apply color correction to image to image you want to make a check mark on that and then you have here the noise multiplier for image to image now you want to set this to zero but this doesn't work initially so what you need to do here is on this slider you want to rightclick and then click here on inspect now on the right size this will give you this console and in here you have a minimum value you want to write zero in here you don't need to save anything afterwards go up here to the number value right click inspect again you have a minimum value here you also want to type zero in here then you can close the console again and you want to pull your slider down to zero next you want to decide on a Model you want to use in my case I'm using proten version 2.2 for anime and I would highly suggest again that you click on these images here if you see something you like and you copy over the prompt here and the negative prompt as a starting point and then adapt that to what you want to have so in my case you can see here that I'm using the negative prompt but I have highly modified the normal prompt in here so I changed this to Japanese woman in a hotel room I found that you actually get better results with this method when you describe the background so this is why I wrote in a hotel room in my case I also removed all of the artist names and adjusted the rest here a little bit and this is of course up to testing so the first thing you want to do before you actually get started to use your batch is to use image to image and you want to drag one image up here and then the same image down here so these have to be frames from your video and the important part here is that you choose a frame where the face is clearly visible so you can see the effect now for this method to work well I would also suggest that you choose videos where if the face should be clear the face is close to the camera and has a lot of detail in it because otherwise if you have a a full body shot the face will often not look so nice after that you want to set up your settings down here as usual I would suggest to use uler a because that gives really nice results and also doesn't use much steps I would try to get the best results with the lowest number of steps because you have to run through a batch and if the step count is too big you have to wait a very long time for the video to finish in my case you can see here that I'm using a CFG scale of 10 which is rather high but worked well with this specific project and I'm using a d noise strength in my case of 0.2 to stay rather close to the original image in that case I get nicer details and I get more consistent video but of course it also sticks a lot more to the original video for the control net settings I'm using of course you want to enable it canny and then the model also canny the weight is 0.2 in my case and in my case I chose scale to fit after you found a prompt and settings that work well for you you want to keep these and you want to use the same seat that gave you that great result you can actually click down here on the recycle button to copy that seat over into the seat field here next what you want to do is to click here to remove the image up here from that area and also click down here to remove it from the control net this is really important because otherwise you might get ghost images especially if you have the image still down here in the control R net area now here's how to set up the batch go back to the folder that you have created and you click up here in the address bar now everything here is selected control copy to copy that address and you want to put the address here into the input directory you want to do the same thing with the output directory but then you want to go here for this reverse slash I'm not exactly sure how this called in English and then you want to just write a folder name for example test 0 too usually this folder should be created when you start the batch after that you simply click on generate and then wait for it to finish when your result has finished you end up with a bunch of jpegs or pngs you can also again use the easy GI page to turn this into a GI click here on the first tab then click here on choose files and select all of the images you have generated click on Open click on upload scroll down and here for the delay time you can experiment but usually I also find it works well when you just enter zero and then you simply click on make a gift and after a second or two this has created a gift for you you can right click on that and download it so you can share it online don't share the direct links because this will be deleted in a short time let me know what you think in the comments thank you very much for watching and see you soon bye oh you're still here so uh this is This is the End screen there's other stuff you can watch like this or that really cool and yeah I hope I see you soon uh leave a like if you haven't yet and well um yeah