DREAMBOOTH: Train Stable Diffusion With Your Own Face And Style Together!

Key Takeaways

hello humans my name is scan your air overload and today's video is going to be absolutely insane because today I will teach you how you can merge your own face in a style at the same time using dream booth and the checkpoint merger in stable diffusion so in this video I will explain everything that I observed after doing all of these different experiments these last few days and then I will explain how you can do this too on your own computer alright let's go so as you can see here I did a lot of different experiments to see which one is the best way to mix a character and a style at the same time and if you think that this video took a long time to make well uh yeah you'd be correct and it's actually my second time doing this video Because by the time I finished making the first recording I noticed that the stable diffusion UI has been updated once again but this in turn actually created a very happy accident that I will explain later in the video as you can see here this text right here this was basically my reaction when I saw what happened now I'm not gonna show you what's below because that's for the end of the video but this result is actually pretty insane and I cannot believe the fact that the best result I got was actually on accident but you need to watch the rest of the video to really understand why it was so incredible now I'm gonna say right now there is not a single solution that is 100 perfect for every style that you want so you might need to mix and match a few things mix and match a few techniques to get the result that you want or maybe use the last results that I will show you at the end of the video that could potentially work for pretty much everything okay so let's begin first of all how exactly did I start my testing I started testing with the bass rhaenyra ckbt file that I created using dreambooth and this model was created in a previous dreambooth video that I will put the link in the description down below if you haven't watched it but basically I trained stable diffusion using dreambooth using pictures from runera from the show the house of dragons and as you can see these are the 10 pictures that I used to train stable diffusion using dreambooth and by using this model as a base it will be easier to see in the future the evolution of the images when we mix it with different style now for each generation I use the exact same parameters in the exact same seed as you can see right here 50 steps ddim sampler 10 scale and then the seed also I forgot to say that if you want to have access to this mirror board I will put the link in the description down below it is absolutely free and you can use it as much as you want so another thing that you need to be aware if you see on the right here on the left you see something very peculiar you see that certain words here are between parentheses if you remember correctly if you watch my settings triple shooting video you won't remember that one of the options that was added in the stable diffusion is the ability to put emphasis on certain words in your prompt and the way you do this is by putting certain words between parentheses which means that when the stable diffusion is creating your images it needs to pay more attention to the words between parent and the more parentheses you have the more stable diffusion needs to pay attention to those words if I remember correctly I think that in the example that I was given there was a breakfast The Prompt was kind of like a breakfast with bacon and eggs and if you put the eggs between parentheses you would have more eggs in the final image and the more parentheses you put the more eggs you will see in the final image and this is the same principle that we're gonna apply in today's video so when you have a character and a style if you want to put more emphasis on the style you can use the name of the style between parentheses so that stable diffusion apply more of that style to the image and on the contrary if you want less style and more of a realistic feel you're gonna put the character name between parentheses so the first option that we're going to be looking at is to merge this base runner model with the Arcane style model using the checkpoint merger option now for those of you who don't know what the Arcane style is it's actually called Arcane diffusion and it's actually a stable diffusion model that was strained using images from the Arcane show and these is the kind of images that you can get in apply the style to your character now do not worry I will show you how you can download it and install it on your own computer later in the video but basically this is the style that I wanted to merge with this version of renera and at first I tried to use the checkpoint merger tool now unfortunately as you can see everything right here is actually outdated because before in the previous version of stable diffusion when you go into the checkbook merger tab you had multiple interpolation methods you animated sum and you add sigmoid and inverse sigmoid but now if you update stable diffusion to the newest version you will only see the interpolation method of weighted sum and add difference that I will explain later on so basically all the work that I did here is actually absolutely useless and unfortunately in a way some of the results that I got using the sigmoid version was actually better than anything that I could do with the newest interpolation methods which kind of makes me a little sad because we definitely lost a pretty interesting option now I'm not gonna go into too much details here since everything is outdated but the thing that I noticed is that if you used the weighted sum method everything was a little bit more realistic if you use the sigmoid method everything was more stylized than the weighted sum method and there was also a little bit more contrast and the inverse sigmoid was very similar to the sigma inversion but was a little bit less saturated the inverse sigmoid was almost like a small mix between the weighted sum and the sigmoid so so right now the things that we have instead is the weighted sum and the add difference now the way I did my testing is that I went to stable diffusion it would be checkpoint merger tab in the primary model I selected the runner model that I created in the previous Jimbo video which was this one the secondary model called B I chose the Arcane diffusion model then I entered the name for the custom checkpoint model for example renera Arcane weighted sum 0.2 from the multiplayer I will either choose the 0.2 the 0.5 or the 0.8 then I will choose the weighted sum and click on run and this simple method will basically merge these two models together and as I said one of the method is called the weighted sum and the second method is called add difference and the add difference to be able to use it you need to put a third model which is actually pretty cool because that means that you can basically merge three different models together now I did not start doing this because this would really open basically an infinite amount of variation so instead in the third model the tertiary model I would choose the exact same as the first model because when you actually look when you over your mouse on the add difference button you can see that the formula used is a plus between parentheses B minus C multiplied for the multiplier meaning that if you put the exact same model here and here this would actually amount to zero and you don't want that so instead in the model C I will just put the exact same as Model A and this is already in this test right here so first what did I observe by doing this test well first of all that the checkpoint merger 2 is actually pretty cool because it does indeed mix the two models together but that also means that you are losing information on both sides meaning that the file little character will not look exactly like the base model and that the style that was trained will not be 100 percent represented in the final image so by actually merging the two models together you're not exactly creating a perfect mix of the two models basically imagine if you add Model A that was represented by a letter a and a Model B represented by model B you might think that if you merge them together you might create a brand new model that is at the same time A and B which both have the same amount of information as before but just in a brand new model that is like twice the size but in reality it's almost as if we're cutting the model A in half and the model B in half and we put them together it does create a brand new model when we only have half the information from Model A and have the information from model B put together to create some sort of frame Stein's monster so the checkpoint merger option is never gonna be perfect you are gonna lose informations from both models so basically if you use the 0.5 with weighted sum and the add difference method it's almost like taking half the information from one and adding half the information from the other and as you can see in the final results you see a small mix between the base runner model and the Arcane style model so if you use only 0.2 with the weighted sum method it will take 80 percent more information from the character model and 20 from the style model meaning that the final image will be more realistic and on the opposite side if you use 80 percent with weighted sum you're gonna have eighty percent of the style and twenty percent of the character meaning that it's gonna look more stylish but it's gonna look less and less like your original character now in this case the add difference one is actually pretty interesting because even with zero two with a difference you actually have a pretty good mix of both the character and the style and with 0.8 with ADD difference you have even more stylized images than the issue where the character doesn't look like your original character is still there and if you want to know what kind of image the Arcane style produces alone these are actually the images that were produced using only the Arcane style model so if you put them next to the images right here you can see that they are fairly similar and it makes sense because the more percentage you're gonna add from the style model and the more it's gonna look like a 100 version of the model so this is basically 80 percent of the style merged with the rhaenyra model and this one is 100 so depending on the style just by doing this could actually give you pretty good results in this Katie as always the more you put and emphasis on the Arcane style the more parentheses you put the closer is going to look to the base Arcane style model so if you want to add more style to the final image you need to put the name of the style between parentheses and vice versa the less style you want the more parentheses you need to put around your character's name now this is nice and all and the checkpoint merger 2 is pretty cool but what if there was actually another way what if you would simply use dreambooth to first train a style and then use that style model as a base to train a character over it well this is what I actually done with this example right here the rhaenyra dream Booth Arcane style base in dreambooth I used the Arcane style that we saw previously and I used that ckpc file as the base to train a brand new model and in the end I got this which I'm unfortunately did not exactly work very well it's almost as if the Arcane style was completely erased and replaced by the face of rhaenera and it did not matter how much emphasis I put on the Arcane style nothing would really work so then I asked myself since the Arcane style works with character that were already trained with the base stable diffusion for example if you put Elizabeth Olsen in the Arcane style this is the kind of images that you would get or if I put Christina Hendrix I would get images like this meaning that maybe the best solution is to First make sure that stable diffusion knows our character before applying a style over it so basically instead of first training a style and then training a character over it you first train a character and then put a style over it but here since I'm not the one who created the Arcane style training and did not have access to the exact images that was used to train the Arcane style model I simply use the images that was used to create the mid Journey style that I created in my previous Dreamboat video and you can see these images at the top of the board these are the 20 images that were used to train the mid Journey style so as I said first I used these 10 images of rhaenera on dreambooth to create a model of that character and then I use that character model as a base to train the mid-journey style over that model and this is the final result now unfortunately we also have the exact same issues that we did previously using this method right here as I said previously when using this method it's almost as if we erased the Arcane style model and replaced it with the face of renera and here it's almost as if we we erased part of renierous phase and replaced it with the style meaning that yes indeed the style is now more present in the funnel image but unfortunately the character does not look like rhaenyra so this is also not a very good option so then I started doing a little bit of experiments to try to solve these issues since here we have lost a little bit of the face from the original character that we wanted to train what if we took this model and we merged it with a previous version where this time the character looked more like rhaenyra which is what I did here when I mixed the mid Journey style dream booth and the renira dream booth and as you can see here the final result is actually pretty interesting where we basically had a pretty good mix between these two models to create a brand new one where the character does indeed look a little bit more like renera and at the same time we also have a little bit more influence from the mid Journey style but let's be honest the final result is not 100 perfect but this could work for some of you depending on the style and the image that you want to train and then I did the exact same thing but this time with merging a style since we didn't have the previous linear dreambooth model that indeed did look like our character but unfortunately erased part of the Arcane style that was used as a base so if we take this and merge it with the Arcane style to add a little bit more of that style back what kind of result can we expect well I did an experiment and actually the final result is not that bad well indeed our style is definitely more present than in the previous version as you can see here but unfortunately the character does not exactly look like renera so again we have basically a similar issue as we we have right here that basically inverted here the character looked closer to rhaenyra but had less style and here we have more style but the character look less like grenera so again it's a very interesting experiment that you could definitely try depending on the style and the type of character that you want to train but it's never going to be 100 perfect now as I said earlier in the beginning of the video I actually discovered the method by accident that could basically solve all of these issues in one go because after doing all of these experiments and actually recording the previous video I actually asked myself this question what if instead of first training a style and a character over it or a character than a style over it we will train at the exact same time a character and a style in the exact same training well this is the experience that I did and well this is the final result and the final result is just absolutely insane I cannot believe my eyes when I started creating the images and these was the images that were created basically in one go we solved all of the issues without any of the downside as you can see right now basically all the images not only look like rhaenyra the base character that we wanted to train but are also extremely close to the mid-journey style that we wanted to replicate and that was an absolutely beautiful surprise now of course again it's not 100 perfect but seriously we are almost there and compared to all the previous methods this is by far the best one and the way I did this by accident is that I used basically all of these images together I use these 10 images of rhenera and these 20 images of mid-j Journey inside one single training and I trained this as a person now unfortunately I forgot to record this part of the tutorial so this time I'm just gonna explain to you pretty much everything that you need to do before the training section but if you don't know what to do after these steps I highly suggest that you watch my previous dreambooth videos because the steps are basically the exact same so to do it you're gonna click on the link in the description down below you can arrive on this site called runcloud.io you're going to create an account and then you're gonna select a GPU that has at least 24 gigabytes of vram so in my case I will be donating the RTX a5000 then you're gonna select your template and choose runpod by torch then choose 60 gigabytes of containing disk and volume disk then click on continue and then click on Deploy on demand and then click on my pods then click on this Arrow right here and wait until the Pod is initialized after the Pod is initialized you're gonna click on connect and then click on connect to Jupiter lab then click on this button right here and then copy and paste this command that you will find in the description down below and then run the cell this will create a brand new folder called dreamboof stable diffusion you're going to double click on it and then double click on dreambooth run pod and you're gonna come here select the second cell and then run it to not run the first one and then you're gonna wait until the sale has finished running and you know that the cell has finished running when you see a number right here instead of the star and there you go as you can see we see a number meaning that the cell has finished running and you're gonna scroll down and you're gonna click on this cell and click on run so it's gonna ask for a hugging face token so you're gonna click here create a brand new account on the hangingface.com go to this link right here that you will find in the description down below you're gonna have to accept the terms of conditions and once you've accepted the terms of conditions of this page you will click on your profile then click on settings click on access tokens then you know scroll down and click on new token and then you're gonna create a name for your token change the role from read to right and then click on generate a token then scroll down and click on this button right here that says copy token to clipboard then go back here and paste the token right here and then click on login it's normal if you don't see anything happening just wait a few seconds then scroll down select this cell and click on run and this will basically start downloading the base 1.4 stable diffusion model and then place it into this folder right here and there you go the model that ckpg was successfully downloaded as you can see right here then you're gonna scroll down in the regularization images section then you're gonna select this cell and simply click on run and you're gonna scroll down and here you're gonna upload your training images and there you go now for me I've already done so in a previous video again if you don't know what I'm talking about you should definitely check out my previous Dreamboat videos and once you have all the Imgur links right here so there shouldn't be around 30 links for me 10 images of for an era in 20 images of I mean journey and then you're gonna click on run then scroll down select this cell and then click on run and there you go as you can see the 30 images were downloaded from Imgur and placed it into the training underscore images folder then scroll down and this is the basically the last step this is the training section here you're gonna choose a name for your project I'm gonna call mine MJ ranera for me Journey runera now the max training steps since I have 30 images it should give me a training steps of 3030 but in my case I'm simply going to choose 2020 which in my experience is usually enough You're Gonna Leave the class word as person and here you're gonna choose a token in my case I will choose the exact one as the project name MJ renera and then you're gonna click on run now I'm not gonna do it because I've already done so as I said previously it will take around 15 minutes to train and in the end once the streaming is complete and you see a number right here you need to scroll down click on this cell and then click on this button to run it and what it's going to do is that you will see a brand new folder called trained underscore models and if you go inside of that model you're gonna have to right click on the model that ckpg and then click on download and this will download the file onto your own computer now unfortunately before you could use this option right here to upload this model to the Google drive but unfortunately as of right now these commands do not work anymore so unfortunately your only solution is to download it directly on your computer and depending on your connection it could take some time and then once you download the model onto your computer you're gonna select it Ctrl C to copy it you're gonna go into your super stable diffusion 2.0 folder stable diffusion web UI into models stable diffusion and then you're going to paste the model right here then you're gonna go back and launch stable diffusion then instead of your stable diffusion you're gonna come right here in the stable diffusion check Point select the model that we just pasted you're gonna wait a little bit then you're gonna input your prompt do not forget to use the name of your character that you created previously followed by the name of your token in our case it's person choose the sampling steps the sampling method I'm going to be using ddim just like the previous examples the scale and finally the seed if you have one and then click on generate and this is the final result now one thing that I found that was actually very surprising is the fact that in the model that was trained using the mid-journey images but did not have any mention of the word mean Journey if in the prompt I inputted mid-journey style I would actually usually get better images than if I did not use the mid Journey style in my prompt which is actually really surprising because again as I said the only thing that we used on just me Journey images we did not put anywhere here the word me Journey we simply used in my example MJ renera this is the word of our character remember we trained a character called MJ renera but somehow by using mid Journey style in our prompt we actually get better images I do not understand why but this is the result that I got and again if you want to check any of the images that you saw today the link to the board will be in the description down below so that you can really compare how everything worked for me so in conclusion what if you want to do the same what if you have a character or if you want to train your face and you want to merge your style what option do you need to choose well basically everything is going to depend on the character that you want to train and the style that you want to use sometimes you might get better result simply using the checkpoint merge option which is actually a pretty easy way to do this but the final result might not be what you always expect but this this option is definitely useful if you have not trained a style yourself so for example in our case we used Arcane diffusion this is something I did not do myself I just went on the Young and face website went to two files and version clicked on Arcane diffusion V2 and then clicked on download and then put this model and put it inside the models stable diffusion folder so at the beginning I already had a style that was done by someone else in that case I cannot use this method right here because unless the person provided the base images that was used to generate and create the style you cannot use this method so if you already have either a character model that you created or either a style that you already have or both if you already have trained a character previously and you already have a style that you downloaded from somewhere else you should definitely try to use the checkpoint version option to try to mix these two models together it might not be perfect but that's definitely the easiest way to do it now obviously if you already have images of your character and images of the style that you want to train you should definitely try to use this technique right here when you train a character in the style at the exact same time using dreambooth and maybe try in terms of ratio to use only 10 images of your character and 20 images of the style that you want to apply to the character because this experiment that I did right here this ratio worked for me really really well and there you have it folks now you should have all the information that you need to be able to mix your own face and a style on stable diffusion thank you so much to mp2 supporters who are making these videos possible you guys are absolutely awesome and that being said thank you so much for watching don't forget to subscribe and smash the like button through the YouTube algorithm and I'll see you next time bye bye