I LOVE THIS - Sam2Seg Points Editor + Workflows

Key Takeaways

hello my friends how are you doing today I want to show you how to use segment anything too with prompts to select anything in a picture and then also how to use it with the point editor both of that is really cool but before that I want to tell you why I have a really big problem so here's the story I was flying with 0 Hut Airways and I had to change my flight because I was invited to a comu ey conference in La so first of all I informed them right away that my flight will change and they said I had to call on the day of the flight for some strange reason but we did that even though I couldn't call my friend called and explained the whole situation and they charged him $ 52950 as a fee so I reached out to ITI hat and they told me right away the customer service told me I should have never had to pay this fee this was on the 2nd June 2024 since then I reached out to them again and again and again they always told me I should have never paid that money I should get that money back and for some unexplicable reason they have to call me but they won't call me so all of that is pretty absurd and I just wanted you to know about that situation so you can make your own decisions I might never fly with them again but let's let's talk about some fun things like the beautiful workflows I built for you and as a patreon supporter you can download them right away so the first one is using Florence to run up here you can also see the pack that I'm using so this is comu I tensor op and you can download that pack of course to have this note here and you also have this note over here so let let's explain how that works we're loading an image here and we want to be able to use a prompt word to identify an object or a thing or a person inside of that image so in this case we want to just select we want to mask out the hair and we're going to use hair of course as the prompt word as you can see here this is masking out the hair with some growth of the mask with some blur of the Mask going to show you in a second how that works so as I said we have here download and load Florence 2 model this is also automatically downloading the model for you which is pretty nice this goes into the Florence to run node here where it loads the model it also loads the text input now usually this has a text input in here but I changed this to input node you can do this by right click and then down here you can see you have widget to input or input to widget so you can change it to one of these inputs here uh just to remark that to you the reason why I did that is because I built a little control center up here where we have the image preview of The Loaded image we have the segmentation prompt here which is here and then we have the saved image which is the export or exp output of that and here we have the K sampler prompt for that and also the K sampler D noise I also made this as an external note for the case sampler so that you could control it for from here and have an easier way rather than going all the way jumping around through the workflow okay so let's get back here to see what is happening so first of all we have here Florence to run and what this basically does you can see here we have a note for the coordinates this is just creating a box that is telling you where that object is so here you can see this is object number one which is here and it's in this location this does not create a mask yet so we still have to create the mask and this is happening with Sam 2 segmentation down here so we are sending the data with the B boox down here into Sam to segmentation we also have to load the sem to segmentation model this is also download so it's automatically downloading the model for you and for this also we have to import the original image now this is creating a Mas for us but what is missing here in this note is any ability to grow the mask or make mask blur so we have here this note here from the kchain notes grow mask with blur here I have the expand with the value and we have the blur radius with the value play around with that to your taste to whatever you want to do with that and then we set the image we sent the image over here first of all I'm using fill mask area with tea so this is basically replacing the shirt with a fill in at that location so when you have a lower D noise you can still use that area but it doesn't have 100% effect on that so that can improve the image quality quite a bit if you do some in painting and this is then sent over here to vae incode for in painting so this is using the pixels including the first in paint for Tia and then also the mask on top of that going as a latent into our K sampler now in the K sampler we have positive prompt negative prompt we have a model in this case I'm using 1.5 model realistic Vision not an XL model because it's just easier for most people to use the 1.5 models even if you have a older CPU and you probably also have this model on your drive and then we are using the D noise as I said I have this out side and then of course we need here a prompt conditioning for the control net so up here you can see with the gray point with this dot here I can expand this or I can make it small so here we have the output from our positive prompt this goes here into the apply control net note for the conditioning and here I'm using load control net where I have of course control net depth fp16 for the 1 5 model and then here I'm using the depth anything Noe with the depth anything model that is coming from the comi control nut auxiliary pack and this will create this depth map for you but you can of course if you want to also use another node for the pre-processing of the depth map image so when we have done that of course you can see the image is going in here that is pre-processed from the original image that goes here into our positive prompt and that is then just rendering the image and then decoding it with the VA decode which replaces the hair here to some pink hair and you can see how good that looks there is no Edge there is no problem this is really fitting to his head really beautiful let's go to the second workflow that I built for you and this is not using a prompt this is just using this little dot down here that you can set anywhere in the image now there's actually two ways to load that image first of all you can use the load image note and then put it in here if you don't see that you have to update your notes you have to update it by going over here on the manager you have to click on update all to update all of the packs that you have installed and as you can see here this is coming from the KJ notes pack pretty important now the second way to load the image is by right clicking into this area if you don't see that area you have to go down here and click on new canvas to create that area and one thing that's a little bit strange but this is still in development for that note is you have to manually set the size of the image here so you have to type that in by hand and then you can rightclick you can have here load image to load the image directly into the Noe now you see here you have two of these dots and of course the Green Dot is for the positive prompt and then the Red Dot is for the negative prompt and to select the area you don't want to have basically and these two coordinates that we have here they are going to be sent out because again this is not creating the mask this is just telling it what kind of part of the image to mask but without a prompt we just set it to that area and it will try to figure out what that is so again we load here the Sam 2 model we load the image and then the two coordinates that we want to have and this will create the mask for us and again I'm using here the grow mask with blur note from the KJ noes pack with the same values we get here the same mask but we don't have to enter any kind of prompt we can just like pointed it with this point editor this is what the reason for that is and then the rest of the workflow is basically the same the only difference here is that I have here my prompt as is and also the D noise as is I didn't really change anything about that this case I use 0.97 but basically if you have a depth map you can set it to one because the depth map will Define the shape of the object and then in this case I change it to Blue here as you can see it works beautifully it looks like it's this real here it actually has more detail than the midy image that all of this is based off so the whole process is pretty amazing the AI art challenge images for this week so the first one is for the topic music as part of an abstract and realistic landscape and here we have these musical water drops that look like notes with a beautiful landscape in the background you can almost hear the music floating through the landscape that is really cool the topic for Tuesday was lost civilizations and here we have this really cool temple in the middle of the rainforest we have to explore that are finding that Temple this is kind of the scene of a video game or a movie that you want to see or play all of that looks really amazing the next image is super cute it is characters inspired by music Shores and I do love cats and here we have the coolest hip hop cat of the Block in the house look at that really cool guy here with a gold microphone and the big chains this guy really knows what's hot next for Thursday we have Battle of the elements and this again is really nice fire is fighting water and they're almost looking like they are shaking hands so maybe they finished the fight and made up afterwards congratulated the winner or whatever but it looks really amazing the action is nice it really reminds me of the Gilla movies very cool next for Friday we have surreal still life and this is a very very nice Dage and also very high quality let's zoom in a little bit look at these details that is just amazing where we have these deer with a lot of endlers that change into the branches of a tree with a lot of fruits on them very nice image for Saturday we have the Samurai battle in the cherry blossom season here with two dinosaurs that are swinging their katanas at each other very Scenic battle here with a nice sunset in the background and these Japanese buildings and of course the beautiful cherry blossoms and for Sunday we have animated food because that's the day where you can lash out a little bit on some cake and here the cake is a little bit angry because probably he's going to be eaten soon the fork is already standing behind him so that's what you want to have on Sunday a good cake with some cherries on top if you want to take part on these join my Discord group where we have these challenges every day of the week and maybe you're going to be in my next video leave a like if you enjoy this video subscribe to my channel if you want to see more tutorials like that and thank you very much for watching bye oh you're still here so uh This is the End screen there's other stuff you can watch like this or that really cool and yeah I hope I see you soon uh leave a like if you haven't yet and well um yeah