LLM in ComfyUI Tutorial

Key Takeaways

today we're going to be looking at this workflow we have generated a cyber Punk woman here and the specifically the nodes that we are looking at are these so we have Sarge llm node we have the sarge output node here that shows us the output and what's happening here is that we have input a prompt here so I have input cyberpunk woman that's all I've written I'm loading a model here the mistol llm model and then we have instructions down here so you have generate a prompt from prompt which is this right so after that the llm generates this so we have a grest cyber Punk scene unfolds with a fierce female protagonist at its Center the image features a striking woman dressed in Sleek futuristic attire you get the point right so he takes our text cyberpunk woman and gives us an a long prompt which is what flux loves so specifically for flux that actually is trained on Long prompts like this this is very very powerful so you can just and something you know Fantasy Island and if you cue this up oh and if you don't recognize the user interface here uh this is actually the new comy user interface so if you want to see how that works you can just go into the settings down here and here in the menu you can select uh top or bottom here disabl is the old user interface so instead of our Fantasy Island prompt now we got a breathtaking Fantasy Island emerges as a vibrant and Lush setting in this digital illustration yada yada y you can see all the text here I'm not going to waste your time by reading it all and as you can see this is our image so this is super super powerful what we have done is we have connected from the generated to a clip text encoder so usually it would look something like this right so so this is an old um prompt box so if you want to build this yourself you just take one of these and you you convert widget to input convert text to input and then you can drag a text input into that box there are actually three nodes that you can use so let's sort this by alphabetical let's go check sge llm and then you actually have an advanced nodes options as well so you can drag that over there if you want to change some of this but let's get this installed on your machines so I put together a short little guide up on my patreon it's available for everyone right now there's more content over there for paid subscribers so this link is going to be in the description uh the workflow that we saw this one basically without all the mess I made is going to be available for download so just check the link this one down here and what you need to do is go into your manager so let's do all of this right now yeah so here we have a version without the notes installed we're going into the manager up here you can either select install missing customes if you are dragging and dropping the workflow or you can go into the custom nodes manager and actually search for search llm and here you have it install that there are multiple versions of llm nodes the reason I like this one is it's uh simplified it's easy to use doesn't have a lot of feature creep and well I know the developer a little bit and he's a great guy you also need to uh restart your comy and we also need to download this model right here so this is a hugging face link if you click this you'll be able to download that and then you need to create the directory inside confy UI models and this is the directory lmore ggf so it would look something like this you would go into your confy folder models llm uncore GG UF you need to create this directory so make sure you do that and then drop that model in there and that's basically it then you're done and now if you have restarted you need to actually press refresh so either control r or press this little button up here depending on your browser and now your SAR llm should load we can test this out quickly you can see that the model this is loaded here if it is not make sure it's in the folder press refresh which is this button here or in the older interface it was in this little menu on the side it just said refresh so it depends on which version you're at so let's try mercenary cat we're generating this as it's coming in live now we can see we have this a Gran action-packed illustration presents a menacing feline character who m is the essence of a ruthless mercenary y y y and here we have our little mercenary Cat made with the flux and we're actually using the N4 model it's up to you when you want to use but bear in mind this LM model is going to be hugging a little bit extra of your vram so that can be an issue for people now what you can do if you are testing this and you are getting out of memory issues sometimes you can use uh like a EAS Flex model you can use nf4 you can test one of the lighter Q versions but if that's still not enough I recommend that you go try a Cloud solution you have think diffusion for example just launch app uh we have comy here we actually have if you go to the beta version you actually have an ultra 48 GB of vram uh but from 16 gigs to 24 to 48 so that should cover most needs I mean I'm only on 20 4 gigs so that is no issue at all so that should cover most people from local installation to Cloud solution for the people that either don't want to have a local install or can't because of the ram requirements so I think this is super cool to be honest the cool thing about this is this isn't only you know final thing right you can actually change the instructions so I mean you don't have to have a prompt here you can just say generate a random image prompt and if we do this and we press Q you can see here now we got just a completely random prompt so Whimsical DREAM Act landscape unfolds as an ethereal Sunset bathes the scene in a warm Golden Light so let's see the subject is a group of Fantastical creatures among clouds and wild flowers let's see what that looks like so here we have a completely random prompt and we can just go back and we can change um this to whatever we want because this is an llm right so this can take any input write me a short novel about a parrot and if we do that it's going to write me a short novel about a parrot now it has a max tokens here so this is what you can change for the length here we got the title feather taals the Enigma par so I think in our image here we're actually going to get a title but uh let's see about that so you can treat this as your own little chat jpt now we didn't get a title actually in the image but we got a little parot here so that was unlucky but uh it is what it is let's type here octopus with socks title above reads sock socktopus let's see if we can uh get something out of that in an imaginative and whimsicle illustration titled socktopus an octopus is depicted sporting colorful quirky socks so let's see if we get that or it doesn't seem like we are getting that so let's see if we can generate the second time so we got an octopus that is wearing well sort of socks I don't know did that work I don't know what socks on octopus octopi would look like anyway um want to take a moment and remind you guys that we have in my Discord or our Discord the community Discord the weekly challenge that's been going on for 86 weeks now so we are currently on the 87th challenge and these are some of the winners from previous challenges if you want to take part in the challenge just go into the Discord all the links are going to be in somewhere on the channel description or the video description too I guess and every week there is a new challenge and a new winner and let's check out this week so we have this week's challenge is the humanitarianism art and media contest so it runs currently from Wednesday to Wednesday so go check that out it's aailable in Discord and the link in the description as always thanks for watching I'll see you in the next one