Flux ControlNet - How to guide for ComfyUI

Key Takeaways

today I'm going to teach you how to take this and turn it into this or this and this and this and this will you get the point and this I'm going to teach you control net and this is not just any control net video because there are a lot of them out there and I've made a few myself this is control net for flux specifically and there are also many control Nets available so which one should you choose are they different from stable diff Fusion control Nets yes yes they are will the results be better yes and know but I'm going to teach you you're going to get a free workflow that you can use and also full guide to how to get started so let's get on with it oh and what do you call seagulls to fly over the bay Bagels so today I'm going to show you this workf flow but don't be scared this is fairly simple it's not as daunting as it looks so what you got going on here is you loading an image here we have this woman on this island here in the background we're setting our size of the image we are loading our models so we are loading well flux here specifically we are writing a prompt here in the Second Step so we actually have the steps here load image write prompt select control NS press Q fairly simple right so we're writing the prompt here in this case we have woman fashion model in pink dress background is magical fantasy world and then we have a bunch of control Nets so this is where it looks kind of messy but we actually it's fairly simple so you can enable and disable control Nets like this and if you've seen my previous controller video for stabil Fusion this is the sameish workflow just adapted to flux but uh with with a few changes so if you want to run cany for example you just select here and oh cany has been enabled now this is Will caveat and when it comes to flux control net there are lots of models available well not lots but there are a few available but there are no perfect ones but what I did is I've I'm going to give you you want to use so instead of having like 1 2 3 4 5 six different models to download you're going to download one and this is for ease of use right now whenever there are six different great controlling models you can just download those and put them in here separately but right now we're using one we're getting into here this so this is the output this is where the generation gets done and then we have an optional upscaling step so this will be disabled by default but if you want your image to be twice as big or bigger this is where you enable that so let's go back to the start and let's go over this right so we talked about loading an image here so this is where you're dropping it in and if you wonder where can I get this beautiful workflow there's going to be a link in the description it's also from here right if you go to the guide it's a free guide don't worry about it don't be scared about the link if you scroll all the way to the bottom Flex control net and once you got this up you need to go into your manager install missing custom notes it's going to be a bunch here I have them installed but you're going to have one here two here maybe three here and just click them or select them all here and press either install here or install down here your setting size currently 1024 x 1024 which is the flux default I have a handle resolution table here so if you want to see the default resolutions for flux and the aspect ratios you can pick sizes from here if you don't want to do it by yourself we are loading a ggf flux model where do you get that well glad you asked my friend so you're going to go into your manager either up here or if you have the old UI it's going to be on the side here somewhere and I can actually show you the old UI so if you're using old UI it's going to look like this and then you have the manager here so this is the manager it's say back to the new UI uh so what you're going to do it's go into your manager if you don't have the manager I recommend that you um follow my how to install comy guide so just search for how to install comy Sebastian camp and there you go model manager and here you can actually search for uh flux Q8 for example and here we have flux Dev Q8 and you can just press the should be see you see there's an install button here and there's a little green check mark here and that's because I've already installed this so you can just just press install this uh if you have a good GPU you can use the Q8 model so when it comes to ggf and flux models they range from q1 to Q8 Q8 is the best one so the beefier machine the better model you can use if you're not sure maybe start with like Q5 for example just make sure oh there's a lot here well make sure that you read what's going on here you want the dev not the schel so we want the dev one and then you can just pick that the top one here that's fine but for this uh I'm going to use the Q8 so I have that installed and you're going to do this same for the other right so uh the ve is AE do save tensors so where do I find that Seb is it in the model manager again you're a smart one so AE do safe tensors you don't have to type all that out but you can actually find it here AE do safe tensors and it's a little green check mark means I have installed it and you can too if you press the install button there is here for you and the same goes for the other models as well so you have the clip L and the the T5 XXL here you will also have download links if you prefer to do that way so if you follow the link in the description you're going to get to this page Flex control net ultimate workflow and here you have text and images of how to get everything running right you remember this from the the video we're just doing yes it's the same one and this one Dev Q8 yes it's the same one a safe tensors yes it is the same one and those can be found in the links here and also where to place them if you want to do that manually so two ways to do that if you are loading a lower-end q model then I recommend that you also load a possibly a lower end text encoder what does that mean well this is fp16 there's also fp8 here I have one called fp8 here's another one called fp8 just have one that's named T5 fp8 well actually these two should be the same you can find them in the model manager and also in the links if you download them and you can't find them in the list here you're clicking here and justes nothing well you have to click the little refresh button and that one's up here so if you refresh you're refreshing everything and then you should be able to find stuff in your list if you have downloaded them if you're trying to run the workflow and this turns red this this is wrongly selected so just go to this red border node click here select one of the models and try again the reason for that is that well maybe when I created this workflow or someone else created this workflow they might have saved their models in a different folder than you have so that's why you may have to go in here and reselect them we have optional lauras they are disabled by default so if you want to use them you can just go here right click bypass control contr b or command B it says here optional so this is if you know what you're doing if you have no idea what this is just leave it as it is you're going to be fine okay we have got most of it installed you might want to write a prompt I have woman fashion model in pink dress background is magical fantasy world we also want to load this control net model and oh here's a lot of them this is all the control models I have installed but you're just going to need one and that is well actually it's it's this one control n Union Pro safe tensors but I've downloaded this one manually and this one from the Mobile manager and that's why I have two by default this one is loaded and that is from the model manager so if you go into the manager again model manager and search for control net Union Pro you can see here that you have control net Union Pro two options how are they different well if you remembered from our previous talk there was an fp8 text encoder and also an fp16 and you can see here that fb8 one here is smaller than the other one so if you're on a lower end machine you can download that one if you're on a high ear machine you can download this the big one good the good thing it we just need to load one of them uh no M matter how many controls we're using so get one of them and then make sure it's loaded in here and then you are basically ready to go except for this last step if you want to upscale there is is an upscale model here and I'm using this the noal one if you click here you're going to see all your available upscale models if you want this one specifically this one is not available in the model manager you can find this in so if you just uh contrl F noos you can find you can download nmos subscale model at this link put it in your models ESR Gan folder and that should have all of the installation finished and what you can do now make sure that you have refreshed and all the models are selected you have loaded an image you have written a prompt you can press q and you will see your generation taking place and here I have a different image it's still a pink woman well it's not a pink woman it's a woman in a pink dress and also pink hair here but let's talk a little bit about the settings so there are not a lot of settings that you need to worry about because as this workflow is prepared it will work out of the box with the settings provided but if you're not new to control this if you've been using controlling before you might want to look at at these settings and the strength and the start and the end percent so the strength that is how much weight or how powerful this controll it should be this is slider between zero and well a lot usually people use zero uh and one one is the default but when it comes to this controller specifically if you're running a high strength it's going to break the start percent here that's when the start of the control net starts does that make this sound confusing well it's kind kind of logical so when you are generating we are generating now for 20 steps if you're starting at zero well you're starting at well zero frame zero frame one frame two you your control is enabled and then you can end your control net at a certain point in time so if we set this to one we are running control net for a full duration the full 100% of those 20 frames and you might think well that's great Seb well it was previously in stable Fusion when control nits worked great they don't yet influx so if you're running this uh it's not going to look amazing and I can show you why so if we are generating this now and you will see it coming in here live we are now running this for 100% it actually that's weird Okay so let's increase the strength right oh let's set is to one which is uh default powerful strength so we're running this for strength one 100% of the generation and you can see oh you might be looking here and say oh it works we're uh in the preview here we're seeing the hand and the woman looks to be in a good position and you're correct she is but once we get the final result here in a second you will see oh what's this mess this is not what I'm looking for right and that is why with these control Nets you have to be very specific with your values and that's why we're running this only for 20% of the generation eration and then we're letting flux take over so to speak so in the preview here we can see okay control net is working and it's running for 20% of the duration and then we're kind of dropping the control net we're stopping the control net and flux is taking over and cleaning up the image so to speak so this is a a hacky kind of solution but it works for now and this is the way you have to do it for now until we get better new control L and compared to the uh the old result you can see this one is much much better right and depending on what controll and model you're using these values might be different so I put a little note here if you're using the instant X un Union Pro let's actually put in Union Pro around strength 7.6 we had 6 now n% point2 that's going to give you a a good time if you're using the xlabs model not in this tutorial but if you're using it I recommend a strength of 0.5 and n% of 0.5 that worked well for me and and I also have a little note here F controlers aren working as inted and currently as high values break the images and we also not using xlabs and in instant X nod free of use what that does mean if you're testing a lot of other control net workflows you might notice I'm using few custom nodes and the same goes for all of these mod stries so let's say that we using the open post or you can see here we're actually still using the same control net Union Pro we downloaded from the model manager and we're using a strength of 0.7 here and an m% of Two And if we run this we're going to get the preview of your open pose and well you see in in a second the result of of our image but some smart viewers out there might be saying well Seb I saw that there are custom nodes from xlabs and shacker and instant X where you load this control net and then you select open pose or canny or depth or whatever and yes that exists but I built this for ease of use if you're getting the end result that you want why complicated right so this works with less custom nodes less models I kind of recommend that you use this for now until better controlling models are available and it's as easy as that enable disable and if it's not working the way that you feel it should be working well you can enable multiple control NS if you're getting the results that we saw previously where the images are broken will use multiple control Nets and lower the values because they kind of add up not as much as you know 1 + 1 each equals 2 but sort of 1 + 1 equal 1.3 you know I'm just making these numbers up but you get the point right and this doesn't have to be a woman it can be anything right it can be a building it can be this um profile picture that uh conf uses on his GitHub so if we put in cat hybrid fantasy creature with long ears in a magical forest we're going to get some something that looks like this right with the long ears and um well let's see actually this is just a random generation so let's see what we get you can see from the preview we're getting the ears I don't think we got the the cat kind of creature of oh we kind of did here we go let us see as soon as this is finished I really should have prepar this right but uh it worked out fine even in this kind of anime style even though we didn't prompt for that so that's cool so you can put in whatever architecture cars objects subjects whatever you want there's also a little trouble shooting selection here we talked about some of the stuff model selection noes turn red make sure your models are in the right folder refresh reselect image is weird or ugly turn down controller strength or end percent workflow crashes or goes out of memory probably a lot of you will have out of memory issues so load an fp8 controll and Union Pro load fp8 text encoders loader smaller flux Dev G ggf you know like talked about q1 to Q8 and that should be it you should be able to run flux control Nets within comy as easy as possible thanks for watching hope you learn something as always have a good one see you