How to Run Stable-Diffusion using TensorRT and ComfyUI

Key Takeaways

here's how to run stable diffusion using comfy UI with an incredibly powerful inference engine made by Nvidia called tensor RT this is what we'll be comparing against when I cue The Prompt again we can see how long it takes as you can see the bar you know what is that about seven 8 seconds or so the speeds so yeah much faster much much faster hi I'm Carter I'm a founding engineer at brev dodev and today I'll be walking through exactly how to do that if you follow along with me in the link below there's a link to what we call a long launchable a launchable is really any way to package up both Hardware in this case we're using an Nvidia RTX a6000 this actually also works on 490s a particular container and then some software package it all up into a link so that you can click it below and actually run the exact thing that I have right here as I mentioned we'll be using comfy UI and tensor RT today to generate some fun images this was actually posted by envidia AI on their LinkedIn uh profile with called Nvidia AI where they posted a comfy UI workflow generating essentially superhero poses uh from an image we won't be doing all of that today but I'll be walking through how to set up comfy UI in using tensor RT so all you have to do once you create an account on brev is just click deploy launchable and what we're actually doing behind the scenes now is going and spinning up that Nvidia RTX a6000 GPU and then we are putting the container this notebook on it I'm not going to be walking through the notebook too much today like I do in the other guides because really this notebook is just setting up the environment to start running comfy UI uh essentially what we'll be doing today is using stable diffusion to generate some images based on a prompt that we want but we'll be using tensor RT what tensor RT is is it's a essentially a way that Nvidia created to package up a model into an engine so that inference can be really really fast inference is when you make essentially a query to a model and you get an output in the case of image gen you're saying hey this is prompt turn it into an image traditionally that especially for image gen can be fairly slow and so what tensorrt does is it goes through a bunch of simulations trying to solve the underlying math and actually optimize it for the underlying Hardware so it detects that we're on an Nvidia RTX GPU and then it'll create an engine so that inference is much much faster and we'll actually see that in the demo uh if any point when we want to see where the launchable is deploying I can click go to instant page and as you can see we already have the instance up and running and the container itself is already built any second now we should be able to open up the notebook and we'll be able to start running the cells to set up com UI so that we can actually begin the image gen uh this is a pretty fun demo and as you can see this particular instance is only about 56 cents an hour so if you wanted to you know do this all you could probably do it within you know for a dollar or so as you can see we now have the open notebook I'm just going to click that and what that actually does is it will open the notebook that's actually now running on the Nvidia RTX 6,000 behind the scenes here is The Notebook here and uh like I said I'm not going to explain too in depthly I'm hoping this uh video here is is fairly short but all you have to really do is just click that two that these two arrows right here which will essentially run all the cells you do have to restart the Kuda kernel um I'll briefly just talk about what this is doing it's just basically setting up the environment for comy UI we have to install comfy UI we have to download the stable diffusion model and set up the uh essentially tensor RT to be ready to start accepting stuff um and so then we'll be able to this is what comfy UI looks like by the way comfy UI is essentially a guey or a graphical user interface to create complicated workflows like on that LinkedIn post I'll put this post down in the description below as well and you can see the demo where an Nvidia employee essentially created this workflow to turn himself into a superhero so if you've ever seen the applications where it's like oh create your AI headshot and it'll turn just a random photo of you into like a professional LinkedIn photo this is ultimately what's happening behind the scenes and we'll be able to do it today for again like a total of a dollar and so you can imagine the implications of creating an application like this and uh I highly suggest you check it out like I said all of this will be in the the description below and you can follow along let's see what sell we're on so if you're unfamiliar with jupyter notebooks you can see we're still installing some stuff uh and just essentially it'll take a couple minutes for us to set it up so I'm just going to come back back to you once this is all set up but we'll know that it's all set up once we get down to this cell here and once it is already running so I'll get back to you once that's done by the way if you're curious what running com UI on tensor RT on brev actually means our incredibly talented engineer Anish matot wrote A Blog that goes a bit more in depth into what comy UI and tensor RT are I will also put this link down below and uh yeah it should give you a bit more context on what we're actually doing today if you're new to the brev Channel please like comment subscribe we find a lot of our users through this medium and we want to make sure that you have the best experiences possible we have a Discord as well where people are very interactive with us the the brev team and people like to help each other out I'll put that link in the description as well so join the Discord and if you have any questions or feature requests please leave them there all righty now that all those cells have run and we've gotten to the cell which is essentially just opening up the port and the comy UI server we can now get ready to actually open up the the comy UI UI so once this cell is at this stage it this took me about 5 to 10 minutes what I can do now is go back to my instance here on brev and click access and because on this particular launchable we've already encoded it such that this port is open all you have to do is click this link by the way if this says unhealthy there's a known bug right now we're trying to fix it but it probably still will work if you just click that link and now this is what comfor UI looks like if you uh have never seen seen this before it might be a little bit daunting but I promise you it's not it's essentially each of these nodes is just a different stage of what's called a workflow for image gen and we'll be using the stable diffusion XL turbo model as part of this right now we are not using the tensor RT engine that is will will be built for this particular model so we expect this to be kind of slow and so for example this particular prompt is just beautiful scenery nature glass bottle landscape purple Galaxy bottle I'm just going to change that and instead today we'll be having a little bit of fun I'm just going to say a handsome Lebanese man making a coding tutorial all you have to do then is Click Q prompt and now what it's doing is it's actually going through the workflow and this green little outline means that it's on that step loading the checkpoints will take a little bit but that only has to happen once this is actually loading in the model checkpoints to then actually run inference and so as I said inference is essentially when you actually ask a model a question in llms that would be you prompting it like in chat GPT and then it gives you a response in image gen that might be for this particular case clip is just encoding essentially text into the same space that images live as far as the computer understands right it's actually a long Vector uh Dimension space and how stable diffusion actually works is it takes it basically Den noises an image if you imagine if you put every pixel in an image just ARR random pixel with random colors and then it'll actually start to denoise it towards what it thinks you want based on the prompt and so there we go and as you can see here we go we have our handsome Lebanese man making a coding tutorial uh we'll actually be optimizing for a batch size of four so this is essentially the Baseline this is what we'll be comparing against when I cue The Prompt again we can see how long it takes as you can see the bar you know what is that about seven eight seconds or so and then we actually get the the four images of the handsome Lebanese man making a coding tutorial if we can also get rid of the text Watermark here try it again and yeah probably about 5 seconds or so right and there we go here we have our handsome lemones man making a coding tutorial we could do another one like uh Mickey Mouse in front of the Eiffel Tower how about that and so like I said you can create complicated workflows here where you actually mimic certain Styles or certain poses like in that Nvidia LinkedIn post here we go now we've got Mickey Mouse in front of the uh Eiffel Tower here if any point you want to save these image you can click open image and save the image or just save the image directly pretty good stuff but yeah that so that's the Baseline um it takes about you know six seven seconds to create those images you can imagine if you have an application where many people are trying to create images like one of those headshot apps then the speed at which you can generate images is really important and tensor RT what we'll have to do now is if if we go back to this notebook and we download the build here all you have to click is do download right here and pick a file and then do the same thing for the create I've already done that so what I'm going to do back here is go in here go back here and click load file here and then this workflow is really just building the engine so I'm going to click Q prompt again now building the engine can actually take some time but you only have to do it once and again we're we're hyper optimizing essentially this model we're turning it into an inference engine so that it can be really really fast for the type of stuff we want to do in this case it is batch size of four with images of 512 x 512 for this particular module model which again is stable diffusion XL turbo and what Nvidia has done here is essentially created a really really cool piece of software that will do a ton of forward passes and Hyper optimize in inference down to the hardware level down to the often times What's called the kernel level uh because ultimately what gpus do is they do a lot of parallel programming right a parallel compute and there's a lot of math that needs to happen when you're actually executing for example inference on a generative model in this case stable diffusion and because Nvidia has so much of the software stack they've done it to where they can actually find the optal math the optimal methods to use based on your hardware and so you will get the quickest inference speeds pretty much possible um when you actually create a tensor RT uh engine based on a Model based on your Hardware so if you were to actually move this particular engine to a different piece of Hardware it might not be as fast because again it's been hyper optimized to that specific piece of Hardware which it detects automatically by the way this is uh used on the RTX series of gpus which actually many people have consumer RTX so if you have like a 4090 or something GPU in your PC you would be able to do that you'd be able to create a model to actually run stable diffusion locally in this case I don't have that so I'll be using brev to get an RTX 6000 you do need a certain number of vram um but yeah so if at any point you want to see how long it's taking to build the engine this particular cell we can see that it's starting to actually build the engine again this only needs to have once and then we'll load in the engine and do essentially the same demo we did before but this time we expect our inference speeds to be much much faster so I'm going to wait while this builds I expect this to take another six or seven minutes to actually build the engine all righty so as we can see it took about 323 seconds or 5 minutes or so to build the engine so if I go back to comy UI you can see that we have no more stuff in the queue nothing's highlighted which means we've actually built the engine now so now what I'm going to do is I'm going to click load again I'm going back to down loads and I am just going to pick the create one that we downloaded and you see we have the same ex workflow essentially but this time we have uh the tensor RT loader here and so I'm going to put the same prompt a beautiful Lebanese man making a coding tutorial I'm going to click Q prompt so the first prompt will take some time because again we need to load the model and then load the checkpoint but after that it should be very quick when we actually do that so if you recall when we were doing the prompts before uh on the base uh on the base stable diffusion model it was taking about five six seconds to create four images or a batch size of four let's see how quickly we can get it now all righty it's now executing the prompt and that was pretty fast and here we go now we have the the Lebanese man making the the coding tutorial let's try another one what did we do last time Mickey Mouse in front of the Eiffel Tower Mickey Mouse in front of the Eiffel Tower and so let's see how the speeds so yeah much faster much much faster and here we go we've got our Mickey Mouses oh there's two Eiffel Towers in that one these are pretty good um let's do one more uh at brev we have a SF is dead shirt let's do a cartoon skeleton riding a wave you can have a lot of fun just messing around with this there we go look at that pretty cool stuff so that's really the demo here uh really what we did today is we leveraged the power of tensor RT for stable diffusion and comfy UI on the platform that I work to develop brev dodev uh if we go back to our instance here on brev we can see that we started this instance at 4:13 it is now 4:41 p.m. so this was only about 30 minutes or 50 six sents of compute used once you're done using your GPU I highly suggest you delete your instance it will auto terminate after you're out of credits uh if you have any more guides that you would like us to do then please leave a comment below subscribe I have a lot of fun making these and uh if you're interested more about tensor RT check out either the blog post or the Nvidia AI LinkedIn post thank you guys so much for watching and I will see you all in the next one