AI Inpainting App Tutorial - New Inpainting Stable Diffusion Model

Key Takeaways

welcome to one little coder in this table diffusion tutorial we're going to see how to use the recently released in painting model with your in painting tool in this tutorial I'm going to show you how you can use this grader tool the latest in painting model first let me show the demo I've got an image here and I can use the brush here and then I can select the size I can select the mask as well so I'm going to select this and I'm going to say I want to have checked formal shirt for interview and once I click submit you can see that it is going to use the mask and it is going to in paint that particular section of the image with whatever prompt that we have given let's see how it is going to perform as you can see it has actually put a new shirt um I don't know if it's a formal shirt but it has actually put it short and you can also see the details like the hair is proper and other other things are good so this is a really good model and this model has been recently released by Runway if you want to use this model in your in painting application I'm going to show you step by step what to do the first thing that you have to do is go to the model page this is a model page which I'll link in the YouTube description you have to first go here and then accept the terms of services that's very critical thing because this model is not completely open for everybody to use so you need to have a hugging face account and you need to accept the model's terms of services first once you do that then come back to my notebook which will be linked in the YouTube description open the notebook in into your Google collab environment first check if you have got the GPU by doing Nvidia SMI if you do not have GPU access go to runtime change runtime and then say select GPU if you have already got GPU or if you are using my collab notebook you would already have a GPU so you don't have to be worried the next step is to install the required libraries in this particular case we need diffusers Transformers ftfy radio once you install all the required libraries then you need to authenticate this notebook with your hugging face token the way you do it is first you go to your hacking face account click your profile click settings once you click settings you're going to see access token and then you can copy the token from here and then come back and paste it here when you run this particular code and that is going to save the token in the root and once you do that you have to import the required libraries after you import the required libraries here's the main important step if you have ever done in painting before previously you would have imported something called diffusion in painting but you don't have to do that now now hugging face has merged all these things in the same pipeline so all you have to do is from diffusers import diffusion pipeline once you import diffusion pipeline then you can use diffusion pipeline from pre-train and select the model that we just saw the runway ml stable division in painting model is a specially released model for in painting alone so if you are somebody who is doing a lot of in painting in your stable division application or if you are building a startup based on in painting then this is a model idle you should be using because this is a checkpoint fine tuned specifically for in painting once you do this thing it's going to download model files and after the model files are downloaded now we can build our gradu application which is a full stack machine learning application for Instagram painting with Sketch on it we are going to import gradu as gr and then we have to create a simple function the function is going to take an image dictionary as an input and it is also going to take a prompt which is text as an input the image dictionary is going to come from the gradu application where when you create a mask using the sketch tool on gradu it's going to send two images one is the original image and the second one is the sketch image so that's why we are getting the image dictionary and then splitting the image into two which is my image and mask and we're resizing it and we after we resize it we are going to send it to pipe with the prompt with the input image with the mask image we just created and we're going to ask for output if your image's output is not great or if you it is if it is not expected to the level that you expected I would recommend you to change to parameters one is strength something that you should play with this is more like a trade-off between the text prompt and the input image and the second one is you need to also play with guidance scale so play with these two parameters that will actually give you much better input than using the default value the next thing is once you have the output ready then you can return the output images of 0. so at this point you have the core engine of the code ID the next thing is you just have to create the user interface the way you create user interface using radio is quite simple gr dot interface GRS how we have the Alias with which we have imported gradu and call the function which is the function that you just created and the title of the application so the input is 2 1 is the image where you can say you know upload and the second one is the text box where you can enter the prompt and once you have these two ready then the output is an image because you are actually returning an image here so just quickly summarize the UI you have a title here you have input you have an output and you have a prompt text input as well so you can see that you have a function that's a core engine then you have a title then you have input which is two one is the image itself uploading second is text prompt and the last one is the output one important thing to notice is when you have an image you need to select a tool sketch because that's going to give the capability of sketching on top of the image which is ultimately what we're going to use to create a map create an image mask so at this point your tool is ready you can go ahead and then click upload so I'm going to show you another image and then I can click upload after I click upload I can select a particular section of it so I can say this and I can say blue color lipstick with Glitters so I can click submit and it is going to use this image and take the part where we have created the mask and then it is going to try to apply the prompt that we have given based on the strength that we have asked and then it's going to create an output image you can see on Google collab um on a Tesla T4 machine it has taken uh just about like um 12 or 13 seconds this is this doesn't look good but uh this is a point so another image I I'll go back to the same image that I used you can change the brush size brush size here make make it slightly big select whatever you want I can go select this entire thing and then I can say a holiday shirt or a leather jacket a great color or a black color leather jacket with flowers on it and then click submit and then see how it is going to do one thing that you need to keep in mind is It's always important for you to pick a square image because we are resizing the images squared so if your image is not Square it's going to resize it in a very weird shape that you may not like it for some images it should be fine so yeah you've got you've got a jacket and you've got you've got fluid on it so this is this is exactly the point this in painting model is really good and I've also heard that this in painting model is really good for out painting but overall in this tutorial I wanted to teach you how to use the latest stable diffusion in painting model which is the runway model we just saw you can see here and use that along with your gradu application to have your own schedule where you can sketch the mask and then create a very interesting in painted output just like how we see and Hathaway with a black leather jacket with flowers on it which was not part of what the input images like the input image was something that looked like this and we got the output image I hope this tutorial was helpful to you the notebook and the model link will be in the YouTube description please check it out any questions let me know in the comment section otherwise see you in the next tutorial