Stable Diffusion x10: LCM-LoRA (CODE & Theory)

Key Takeaways

hello Community finally we're going to code today again and today we are here with vision Transformers and diffusion models now you know stable diffusion you know mid Journey our ER image generator our text to image Ai and let's focus on stable diffusion you know we can generate here high quality images from textual description the theor is here the physical process of diffusion so theoretical physics are welcome here and the training of the model is by a reversion of the mathematical diffusion process great you know this now today we're going to focus on something new and latent consistency models which are distilled from the pre-trained Lattin defusion models and since last week we have even something more interesting that is simply fast we are talking about speed we're talking about speed increase from 10 times faster so instead of waiting 1 minute for your image to be generated it is now done in 6 seconds so and this thing is called LCM laa and you say hey finally we have here the Laura optimation also here on an advanced stability Fusion model yes of course so let us start now at first new type of Technology create images from text or simple sketches great what makes LCM special is their speed and there are specific class here of our classical latent diffusion model and they address here the computational challenges by transforming the reverse diffusion process into something more computationally efficient the way this is done is if you want to core Innovation is now theol ability to predict the latent space trajectory of the reverse diffusion directly by bypassing here all the iterative steps and they do this here by conceptualizing the reverse diffusion as an augmented probability flow ordinary differential equation problem and I will show you here this in a second it stand out for its remarkable efficiency and I will show you when we code this you get excellent results just with one to four inference steps so with CM Laur we have now a universal training free acceleration model that can be directly plugged into your various stable diffusion fine tune model or your stable diffusion luras great if you not up to speed here are my videos on the channel you find here about before diffusion you remember we had a variational auto encoders then we had the vector quantized variational Auto encoders here in this video I coded with with you here explicitly the complet thing from the variational UT encoders to unit to clip and in this video explained here the interplay of these three elements how we create the classical stable diffusion if you little bit more advanced beautiful you know there's this stabil diffusion XL model I introduced you here to the control net here of stabil diffusion in Excel and of course the control Laura T2 I text to image adapter for year stabil diffusion Excel and if you are into Jack I have a video here about the TPU multi-parallel processing in Jax of stabil diffusion and we code this in this video so this was the basics let's start latent consistency model something beautiful as you can see again chingua University China October 6 2020 this is something really really nice and there's one sentence in their whole publication I want you to remember they view now the guided reverse diffusion process as solving an outed probability flow OD so those lcms are designed to directly predict the solution of such an OD in the Latin space this is what we're going to focus on right so if you're not up to speed okay one slide for you B this is for you diffusion models you remember diffusion models those are generative models that progressively inject noise caution noise into the data and then generate samples from noise via the reverse D noising process this means diffusion models Define a forward process transitioning the original data distribution to a marginal distribution via transational kernel in a continuous time perspective the forward process can be described as stochastic differential equation you say yipp yes here we also have our standard Brownian motion Now by considering the reverse time SD one can show that the marginal distribution satisfies the following ordinary differential equation called the probability flow OD this is important so this is our baby here this is our beauty that we're going to focusing now in the diffusion models we train the noise prediction model Epsilon Theta to F minus Nu lock QT XT called our score function in approximating the score function by the noise prediction model one can obtain now our empirical PF by sampling this beautiful then for you if you want to Deep dive into this you have to go to appendix a of this publication and you see here a detailed stepbystep explanation by all the step we just rushed through they are about starting at page 12 about four or five pages where they tell you exactly how this is interconnected and how you can compute it mathematically so if you are really into diffusion and consistency models the annex AB CD is really highly recommended to understand this on a mathematical basis consistency model our consistency model our CM model a new family of generative models that enables one step or some few step generation this is the beauty so core idea of cm is to learn a function that Maps one point on the trajectory of our PF o to the trajectory's origin more formally consistency function is defined or can be defined here our F where Epsilon is a small fix small positive number one important observation should satisfy the self-consistency property yes yes yes beautiful and they have here new technique they have here the Skip and the out methodology to introduce you the ORS but remember here our consistency model can either be distilled from a pre-train diffusion model and this is what we're going to focus on or you can train it from scratch so what we are looking now is consistency [Music] distillation this is what is here the key to go down to LCM l so to enforce the self-consistency property you will maintain a Target mod updated with yes yes yes you know of this yes of course but what we going to focus on yes we recall that the PF of the r diffusion process is this beautiful little s here but what I want to show you is here where this Sol us further yeah and we introduce here the skipping step technique as I already showed you and now our LCM aims to predict the solution of our PF OA by minimizing the consistency distillation loss and you notice from our AI system we have a loss function and we minimize the loss function this is it so now we have reduced everything we know about here the distillation and about the consist the latent consistency destillation LCD in the latent space we reduce this to this simple minimization problem yes beautiful this is it and then we have our OD solver up C and you know all of this you know several OD solver but this is now what we're going to focus on great so Ben for you distillation loss is a key Concept in mod model distillation technique used in machine learning to transfer Knowledge from a large complex model you know our teacher model to a smaller more efficient model the student model this is exactly what we do here so we have where is it here now again here we go so we have here a big model where is it Heaven EG where is our model here CM can either be distilled from a pre trained diffusion model so we have here an LCM and an ldm and this is exactly what we're going to do we create now a student model an LCM model that performs as closely as possible to our ldm model while being more efficient in terms of computational resources distillation loss is calculator loss function I just showed you beautiful let's have a look at the efficiency and you remember we have the guided distillation process now this is more efficient and the authors point out here with the previous model the guided distillation here we have a two-stage distillation to support fuse stop sampling yes yes yes important to notice we go from 45 days on an a100 with this new meod LCM to 32 hours on an a100 so wow creating find our LCM is now significantly faster and this is what we're looking for yes yes yes we need to solve now the falling augmented PF beautiful now this is our beauty that we have to focus on and yeah it's the same as I just told you great so now we have done our homework now we know where current research is and now let's have a look what happened last week that suddenly from this point of the concept can now accelerate 10-fold and this is the publication November 9th so G almost uh six weeks later we have now a new publication also by chingua University and now finally we're talking about LCM Laur so Universal stable diffusion accelerator module now this is the problem and they are now based on the Latin consistency model we just had a look at we understand now what's happening and now we optimize this and there's again one sentence I would like that you focus on to understand what's happening here so since the distillation process of our LCM of our Lattin consistency models is carried out on top of the parameters from a pre-trained diffusion model our ldm we can consider latent consistency distillation our LCD process as a fine-tuning process process for the diffusion model and now you say hey we are now in fine tuning and we know everything about fine tuning because the last week the last two weeks actually I showed you in my video everything about here the parameter efficient fine tuning compared refra and then we fine tuned here co-pilot here with a p La a quantization everything here so in our llms in our large language model you know exactly what we do we go from the classical fine tuning to the parameter efficient fine tuning and from the portfolio of methodologies impa we choose one method and this is our low rank adaptation for here our reduced Matrix calculation beautiful so we do now everything from the fine-tuning process of the last two weeks now we go to the stable diffusion process to the L ldm to the LCM and now we apply laa over there this is it it's as simple as can be the complete mathematics has already been programmed for you I know you want to do this yourself I'm so sorry so this is it this is the complete code that you need to use now LCM Laura and this thing is 10 times faster than what you have till the time before last week so you have here my my notebook here I have my pip install Transformer PA diffusers and Accelerate from diffusers we have now our LCM schuer our diffusion Pipeline and you notice we have our stability I our stable diffusion XL base model whenever you see this maybe you have a better base model than 1.0 go with what you have available and then from latent consistency we have now our LCM Laura SD EX sell model now this is exactly what we are looking for so what we do we create our diffusion pipeline from our pre- model yes we Lo now the Lura weights here from our Laura LCM Laura sdxl our schul beautiful we push it to the cter device and then we have here our prompt this is our text we see hey we want to create a close-up photography of an old man standing in the rain at night in a street lit by lamps like a yes yes yes and then we say hey pipe prompt the number of inference stab the Guan scale and create the image this is it few lines of code now you go to hugging phase and you have there our LCM lower sdxl and as I already told you hugging phase decided now to have all the adapters listed now so here this is what I want to show you here wherever you go here to an LCM Laura for for example SXL you have now here indicated hey this is an adapter a Lura adapter for this particular stability stability Fusion Excel based 1.0 model and this is another interesting part that you could choose now your adapter great so let's run this program and in just four INF Fe and step we get exactly this image look at this 10 times faster than sdxl and I think the quality for four inference stops is really extraordinary now if you go hey I want to go to eight inference steps well it's not always that more is better look here here we have 1 2 3 four this is the image I just showed you if you continue with more steps five six seven and eight it depends if you like it go with it but you see even four steps give you two to four steps give you a really good image generation and this is bling fun cost with LCM lower great if you say now I have this model and I want to check now back to my original performance so what you do easy you see here in your pipel hey I unload the lower rats delete it get rid of it go to my schedule load here the classical Oiler discreete SCH so we rever our pipeline to a standard sdxl pipeline by unloading the lower weights we switched it to default thing and then we need much more steps infering steps and so let's have a look here now the outcome if we go from one step four steps eight steps to 50 steps you see just some line of codes couldn't be easier and here you have it so step number one the classical sdxl just noise of course four steps where we had already with low with LC M lower get the perfect picture you see it takes here look look at this horror here it takes 50 50 inference steps till we have a comparable quality so this is The Benchmark here we need 50 steps and with LCM low we just needed four inference steps of our LCM model and this is the beauty how LCM are now so blazingly fast but remember we have to create them from our ldm model and it takes about I don't know was about 30 hours on an a100 to create those lcms so if you have a beautiful model and you want to optimize this model it takes about 30 hours on an a100 to generate your the LCM so you have this once but if you have this model created once you can use it 100 and a thousand times to create here all your images beautiful in this presentation in this scientific paper there's something else that is really beautiful and you will have immediately that you see there's a parallelism compared here to our multi Laura adapters in our large language model and as you can see here we have here a combination of a multi- Laura adapter we have here our base ldm with that a base and then we create here our LCM Laura beautiful this is blazingly fast but if we want to add additional Laura adapters as I showed you in my last video to llms we can do this now here too we create here now a customized ldm Thea Dash with what the or just call here a style vector and the pure LCM Lura they call an acceleration vector and if you have two vectors you can combine two vectors with a linear combination and then you get a customized LCM Teta Dash this is now nice because now we have two Laura adapters that at first we have our LCM but then we have a style adapter let me show you this in code look we combine now our very fast CM Laura accelerator Vector with the style Vector so here we go again from diffuser hugging face defuser is now in the latest version please update to the latest we have now our LCM SCH we have our diffusion pipeline we go again we have stable diffusion XEL base 1.0 we have floating Point 16 yes beautiful we push to Duda yes and now comes the beautiful new Step that we have in addition here we have the lower parameters obain through our LCM Laura training what they call the acceleration vector can be directly combined with the other Laura parameters from a style vector by fine-tuning now this on a particular style data set so let me show you this it is absolutely identical to the llm what we do now here in stable diffusion you load your Lura we have here at first here as I showed you in the last time the Latin consistency they have here the LCM Laura sdxl and we call this Laura adapter whatever you like to call it we call it LCM and then you call here a style model we'll show you in a second or to was the last pen and it's called paper cut sdxl and you give it a name and then you combine the luras with this simple command pipe. set adapters and now you have your first Laura adapter in combination with the second Laura adapter and you define the adapter weights and this is it and your prompt is now hey create a beautiful image a paper cut image and the object is a cute looking Fox great now again as I told you un hugging face you find here all the different adapters for stability AI for the stable diffusion Exel base model and we take now another adapter from hugging phas go there find the adapter you like or create your own adapter and then use it in combination with the LCM Laur acceleration vector and what you get at if you execute this simple line of codes is here exactly you have here cute Fox very fast 10 times faster than the normal sdxl so we utilize here the LCM Laura sdxl model Plus we give it now a style we can now modify it according to our wishes and this style has been fine-tuned here by this organization or person and it is called paper cut sdxl and this is now the fine tune style that all our models will have choose another style model go with fistic go with some sketch illustration whatever you choose you see how easy it is now to adapt this with our LCM and the style adapts so exactly as in the large language model where we have multiple lur adapters now you can do this here also in stable diffusion what a coincidence that it happens now also here on the visual sector okay lurer parameters obtained through our LCM lower training the accelerator Vector can be directly combined with other lur parameters like the style Vector obtained where the style Vector has been obtained by fine-tuning on a particular style data set you have I don't know 100 images or 1,000 images of a particular author of a particular artist and a particular style you like you simply do a fine tuning of the adapter and then for the inference you just click in the adapter and it works it's gorgeous so without any training and please this is what I like to stress without any training the model obtained by the linear combination of the accelerator style and the vector style acquire the ability to generate images of a specific painting style in minimal sampling steps yeah this is the beautiful formulation so what we have compared to the previous numerical PF solvers such as you noce where you have your click down menu and you go for solver now the new LCM Laura can be viewed as a plug-in neural PFA solver that possesses strong generalization abilities beautiful this is the official marketing great now what I find particularly interesting you know we had with stable diffusion Exile we had luras of course it existed before but those were the very slow luras now with our LCM luras our Ultra fast LCM luras we can combine this and this is the unbelievable thing of adapter synthesis look we use here yeah of course latest update code update hugging face diffusers and the latest PA version please update before you do this you can combine your LCM lus the brand new Ultra fast LCM lorus with your regular slow moving stabil diffusion EXL lorus giving them now in total this super power to run LCM inference in just four steps and not 50 steps or 100 inference steps beautiful let's have a look at the code it is so simple again our LCM schedule we go here with our LCM Laura sdxl beautiful you load the weights and great and now you see how do I have to reconfigure the system to integrate here our regular sdxl luras you're not going to believe it there to un hugging phase you remember an adapter for stable a stability AI stable diffusion EX Bas 0 1.0 mile you choose now an adapter for a particular style and this particular style yeah don't ask me why it's a toy face beautiful I don't care you create your own adapter you choose your adapter from hugging face do whatever you like but it is an adapter and we use this standard Laura sdxl adapter that has not been optimized for LM Laura and look we just load here the Laura weights from the is non lcmm give it a name it's called toy beautiful so now we have our pipeline we set the adapters two adapters our Laura adapter our LCM Laura and our non LCM Laura or toy adapter push it to Cuda say hey our prompt is well you're not going to believe it a toy faed man or woman or animal or whatever you can have your negative prompt like like it and this is the command you see just some lines of code and you get exactly here you can combine your if you created your own classical luras for USD XL you can still use them and get the speed increase if you just add here this line of code our LCM low sdxl please note that the quality of This is highly dependent here on your model on your CM Laura conversion here that I just showed you and of course on the quality of your standard sdxl lurus there's a lot of optimization going to happen so please take care that you have the perfect model right yeah as I told you the style Vector can create your own style vector by fine tuning a base model on a data set 100 images from your particular s or whatever you have visual artist present a specific style type of imaginer yes yes yes encapsulate the characteristics yes I just showed you and the pure and the Beautiful power here of our LCM Laura comes into play when we use this combination of the accelerator vector and the style Vector I showed you this is simply the mathematical operation and linear operation and you have some hyper parameters to balance it contribution of each Vector this is identical to our Alpha Vector that we have to augment our lower adapters if you have multiple lower adapters in our large language models if you decide now hey it's time to learn more about multi Laura adapters here can go to my channel I explain here Laura I explain here 4bit quanti Lura I have here rack and path and optimized PA Laura and of course here the last one is PA on Multi Laura expl explained with explicit llm fine tuning and if you want to have here the real the latest here this is my video where I showed you adapted uning with P and L for new llm that there's a new methodology out now but I have also a complete playlist of 13 videos about adapter tuning with PFT and low for language models and now in this video I showed you we can do it now also for stable diffusion and if you're into this here here you will find here the next step that will happen in optimization because it happened here and this is s Laura or scalable Laura and we have here in a multi Laura adapter environment I showed you the theor here in this video I presented you this that here with s Laur we have three significant optimization but not now of the uh minimization problem of the stable diffusion program but here now in the pure compute infrastructure and I showed you here with the s Laura and we have also the multi here in the diffusion model they optimized it here with a unified memory pool for adaptive weight tensors and the key value cash minimize the latency when batching here different Lura adapters and they invented here a new uh tensor parallelization when Computing here our tza multiplication so has now op optimized uh GPU uh Nvidia GPU Coral optimization for a better mathematical operation of the tza multiplication and this has given here in the llm significant speed improvements but this is the next step that going to happen now in the stable diffusion because this implementation of those three measures that gives us compared now to our VM inference a 400% performance boost this has not been implemented yet at a stable diffusion level so you see the research transfer the knowledge transfer from the optimization here from the llms here to the stable diffusion or to the visual has not happened yet but knowing this from the llm side we can say hey and the infrastructure optim iation here from Cuda cornels from unified memory pool for going from the simple page detention here to here a unified paging reduce the memory fragmentation optimize the memory management in total now for our different Laura adapter layers there are still a lot of that can be done and this will be the next step into research for the stable diffusion optimization problem wow I think this is it I hope you enjoyed it you have an idea how simple it is to code this go there take your example just get familiar with it and be amazed about the speed increase for my things and for my Ms it was 10 times faster sometimes even 12 times faster in the generation of the image given I have some simple text expression what I did not test out myself you can also have here an input to the model by simply sketching here some basic sketches and the model will do here a complete formalization yes you can do also in painting here with those Laura LCM Laura adapters so in the next week I think you will see a lot of new example about LCM Laura for stable diffusion but now this is it for this video I hope you enjoyed it I hope you had a little bit fun I hope you learned something and it would be great to see you in my next video