Full FACIAL EXPRESSION control for Stable Diffusion (+Lora Pack)

Key Takeaways

AI understands a few basic facial expressions but for more complex ideas it is harder to get good results just like with the angle and perspective of the face the expression itself with its subtleties and the eye positioning or where the characters are looking at as a first step I highly recommend the use of luras for this which is why I assembled a collection of over 200 facial expression luras that you can download for free in the link in the description of course most of these luras aren't mine so massive thanks to to all the creators that post their trainings to help the community I did however generate a visual representation of what every Laura can do as well as preparated the metadata so that you have the necessary information to properly use each and every Laura plus leaving them ready to click and use I will be using this pack throughout the video but you can use your own if you want and that being said let's move on to the first issue while the positioning of the face is almost impossible to get right with just prompting the solution to it is fairly easy just use control n but but there is a bit of a trick to it normal angles like this where you can see most of the face are easy to get right with just a regular control net reference but I still need to talk about how to get more Uncommon angles like these ones where AI gets a bit confused as you can see in extreme positions the basic Rak isn't enough to make AI understand what we want you could try to help it giving some context with the death map for example and this will get you okay results sometimes but the more optimal way is to use the face open pose model this will get you good results most of the time at least when it comes to the angle of the face personally I use this tools depending on what I need one being just posing myself and using my webcam two I have an open pose rig that does have a face and three a general 3D model that also has a face I also use one other option I export poses from POS my art then I iterate until I get an image that I like not really paying attention to the face itself for now once I'm satis ified with the pose on the character I start posing the face for this you can use any method mentioned before or do as I'm doing right now go to reference angle.com here you have this floating head that you can move around and Poe however you like from this website you could actually try to find faces of people in that position but most of the time I just screenshot the head itself now bring that screenshot or image into control net and use the DW open post full pre-processor this is the best pre-processor by far and should get the post right almost every time if it doesn't you could either retouch the post manually export it in another way or maybe even just use line art instead of open pose we will download the pre-processed image and use an editing software to place the face where it should be like so fill the rest of the image with PL and Export this new control net reference I send the original image to inpainting and import the reference to control n if everything is Right using one the noising strength and a normal prompt should be enough to get get a properly positioned face you can use only mask if you feel like you don't have enough resolution but remember to give AI enough context using a high padding pixels number when it comes to weird perspectives and lens Distortion I haven't found much aside from the usual ways regular prompting can work if you know the name of the camera angle and the lens you are looking for you could also train and use luras for that specific perspective like this going insane one by fallen in kol show in cbdi and also export post death maps and open post Maps directly from sites like post my art as I'm doing here giving me these results now that we know how to position the face in any angle it's time for the actual topic of this video uncommon facial expressions Common ones you can mostly get by prompting or at the very least it will be easy to find Aura made to get them cuz of course a hea has been trained like 3,000 times I've divided these expressions in three different groups according to the method you use to get them the last one being being the most timeconsuming but also the best one starting with facial expressions that can be achieved by mixing multiple oras directly with prompting for example this image with it being the easiest combination possible it's a mix between only two luras and two that affect different parts of the face faceless blush that affects the eyes and T mouth eye that is focused on the mouth just prompting with both luras active is enough to achieve the result we wanted but I want the same clench teeth effect but with a bit more Force to it for this I'll use the excirt Expressions Laura only issue being that this Laura actually affects the rest of the face too if I hit generate you will see that the eyes are now in conflict with the other Laura we don't want any eyes but the scared expression Laura is adding them adding eyes in the negative prompt is helping but not enough so let's adjust the weights a bit first we will move clench teeth the important part in front of the prompt then we will lower the weight of the Lura and its main trigger work so that the effect lowers on the things that aren't the clench teeth part we will also use a higher weight on the negative ice prompt and now we're getting somewhere what if I wanted to add another Lura to this let's see I have changed the eyes to be a looking disgusted Laura this affects the whole face so we will have to solve the conflicting Parts first in this case the main issue is that the two luras are blowing up the image don't be afraid to lower its weights or activate highr fix I will use 0.3 for the disgusted Laura and 0.6 for the scared one and let's see what happens when I add the next Laura to it I will add one of my own luras looking left and it isn't working because there is too much influence from the other two I will just play with the weights of the luras until I get what I want like so problem is that as I use higher values and more luras The Prompt gets lost the simple prompt here was blond made outfit and the outfit is not here anymore if now I were to add another extra Laura it would get even worse also keep in mind that there are luras with stronger bleedings than others and that are less flexible let's look at some easy tricks to avoid this so that you can combine luras properly first you may have seen that I've been using control net open pose this whole time this is because most luras have some bleeding that affects the Bose and having control Nets helps me getting the POs to stay in one place but this can have some bading too which is why I use it with lowered weights and with an ending step of 50% second if the main issue you are having is that some part of the prompt other than the face is is getting lost to bleeding like the M outfit in this case you can prompt regularly first and then change the facial expression with after detailer to make the luras only affect the face for example here I just prompt for the overall image and then with after detailer I get the expression I was looking for sometimes you can try to get the same prompt in after detailer and text to image but using low weights in the main prompt to avoid a heart bleeding while maintaining the main basic shapes and then use higher weights on the after detailer one this makes it so you can lower the the noising strength in case the face is not generating properly if we wanted to do something like this for our example here we could actually try to use the after detailer for just the eyes I will leave the eye detection model in the description but the success rate on this is pretty low so I wouldn't use it for now and just as a fast tip if you have more than one character with different Expressions go to the after detailer settings and activate left to right and then prompt for the expression on the left add in between a squared brackets s e p and prompt for the next one on the right this will inpaint my two different expressions in order from left to right obviously there will be some cases where you can't use after the tailor or don't want to for whatever reason like here where the hand gets completely cut off when I use it in cases like this you could use composable Laura to make use of prompt editing to combine the Expressions you can use it as a way to mitigate the less bleeding onto the rest of the image too for example here we have have the original results for this seat with no luras at all if we were to add those luras without using composable Laura the image becomes a close-up and we lose most of our background so instead I will add the luras after the main image has been decided by typing this here this means that faceless will become a part of the prompt at step 14 it being the first because it is the most important one and has to erase the eyes before it's too late and Hawa will enter later at step 18 with this the image contains a lot more of what it originally was but with the Expressions we wanted even if some parts have been lost the composition is fairly similar with that said another tip I should give you is to try stuff with different checkpoints here I'm using any Laura by Lon because it is very good when trying to combine luras like this but different models will be better for different things okay the second type of Expressions is very fast to explain it's mem or Ultra exaggerated facial expressions you pretty much need to either train aura on them like here and this is probably the best way to do it if you are going to reuse this expression a few times or you could also try with imp painting or combining different control models like bline art open pose and instruct to image but it can be very hard depending on the expression you want finally on to the best method to achieve good facial expressions and with the most control I found and here we can use an extremely powerful type of Laura at its full potential in painting and manual retouching the idea is fairly simple generate a base image and then change each part individually until satisfied this way we can combine multiple luras without worrying about one affecting the other with the only setback of having to do it one by one manually let's go over how to get an expression similar to this one I'll in paint this image right here we can see the slight arcing of the Interior part of the eyebrows it is overall a relaxed expression but with some tears in the eyes and a tight shutting of the mouth knowing this let's look for luras and expressions that have those characteristics starting with sadness which usually makes the interior of the eyebrows Arch upwards I will use this Laura you see that I added as info from zero to three this is because it is an slider Laura and it allows us for much more control than a regular Laura this can be pretty important so I will explain it with a bit more detail after this example for now let's use it as it comes I will first mask the area I want this Lura to affect just the eyebrows and forehead as you can see the effect is a bit too rough so let's lower the L's weight to one and try again you can of course try with different seats to see if you get a different result I will use a higher the noising strength here as I want the eyebrows to be able to move freely if this change results in too many wrinkles like in this example we can fix it within painting again for this try using the method fill with a low the noising strength like 0.35 or so this has taken them out and we have this so let's move to the eyes here I don't don't really care about much just that there are some tears on them and a bit of redness for this I will use this tears Laura also take out my negative prompt and add Tears red irritated to the main prompt and something important I already like the main shape the eyes have so I will use a very small the noising strength less than 0. three in this case now I iterate until I find a combination of Laura and prompt weights that I like this one and don't worry if it's a bit too red on the cheeks fix it like we did with the wrinkles it's time for the hardest part of the expression the mouth and it is this hard because I found a very small amount of luras that are focused on the mouth and none that helped with tightened lips so I will use an anime Laura instead let's go to SD 1.5 now I will use the Epic realism model here we will actually use a Laura that works better with anime but for this example I found it to be quite good moral of the story don't be afraid to use different models and try stuff out I will use the Widemouth Laura with negative-1 weight and the prompt U see now we inpaint the lips and you will see that it kind of tries to give us a tightly closed lips effect we would like the mouth to be wider and a bit more tight though and that will be very hard to get directly so the next trick is yeah yeah it's Photoshop here we actually have the liquefied filter it has a facial feature that can help with minor tweakings like this but I won't be focusing on it too much because some people won't have it also right now it ain't super useful so I will just use my finger and drag it on his face to make him upset and don't worry if it doesn't look great for now we import this back into stable diffusion now lower the the noising strength as we have a better shape than we did before generate again this time it's giving us way better looking images if you still don't like the result you can go back to photoshop once more and repeat the process until satisfied if by the end of the the Imp painting process you have some ugly color variations like here in this other example I made you can go to image to image and generate with the noising strength of like 0.2 or so this should maintain the details of the image while erasing these imperfections and now that you're seeing this there are some extreme expressions like this one where everything is pressed up that are very hard to get you'd probably end faster just training aora for them but if you find a way to create this please share it down in the comments below thank you now before moving into the very last part of the video let's talk about this this is created with a slider Laura I trained it can work with regular prompting but I feel like it's better within painting this is a type of Laura that assigns a different meaning to each value you input as a weight working always at 100% strength instead of having only one meaning and determining its strength by the weight you input like a regular Lura does as you can imagine having this types of luras allows for absolute control but I haven't found a way to train everything I would want next video will be about this topic so I will try my best hopefully there is a way and we can train slider for stuff like intensity like this or even eye movement like this and speaking of eye movement time to see how to make your characters look the correct way you probably have experienced the pain of trying to make a character look a certain way and I just wanting to make eye contact with you and I don't blame it you do be looking helly but sometimes we just want space I have already introduced to you the three loras I made to avoid the looking AB Vier effect looking left right and down it can be helpful situationally but it's not true control so aside from imp painting with loras and hoping for the best what else can we do if we want a very specific eye position for example in this image we would like the character in the left to look straight into the other character while the second one would be adverting eye contact looking down to to the left okay first thing I'm going to do is change his eyes I want some normal eyes please thank you now we got lucky and I could use this image here where the eye is already looking in a nice Direction but let's use this image instead and here comes the part of the video where I tell you to do stuff manually let's be honest this the best way to have full control we have some approaches though the first one and most obvious is to go into Photoshop and either move the IRAs around or just paint a new one another option is to create a reference we do this by tracing the counter of the eye and then drawing where we think the Irish should go like so now at a white background and export in your imp painting tab go to control net and click the upload independent control image checkbox import your new control net reference and select lineer I personally like using a weight of two with control net is more important but that depends on you and at the noising strength of 0.5 of course these are situational and might change for you here you are pretty much done just select the result you like the most if you wanted to add some extra details to the eyes you can inpaint the result again with a very small denoising strength like 0.28 and yes in this case he is looking at her forehead but that's my fault for drawing the eye position in like 2 minutes without paying attention and for her I will use the last method and probably my favorite one in pain sketch I don't know why I did that okay um this is a fairly straightforward approach you just paint the new position directly in stable diffusion without caring much about the result itself since stable diffusion will correct it for you first we erase the current Iris next paint a new one as a rough sketch with a plain color now paint the pubil and let's touch the settings for this to work first activate control net tile I will use a 0.7 strength and 0.7 ending step play with the ending step if you feel like the result is changing either too much or too little then I use the noising strength of one of course you can go lower but again there's no need and finally but very important lower the mask blur to one now hit generate as you can see some results are better than others but just play with the parameters until you are satisfied the important thing is that the eye is positioned exactly where I wanted it and where I threw it I really hope this video helps you create what you needed and make sure to join our Discord server if you have any doubts or just want to share some stuff also you will see that I now have a coffee page I created it mainly to have a place where I can post the resources I offer in my videos thank you all for watching see you