Build Image Captioning Python App with ViT & GPT2 using Hugging Face Models | Applied Deep Learning

Key Takeaways

dear fellow coders welcome to one little coder there is a new image captioning model in the town and that has been released by sachin thanks to sachin for releasing this model and we're going to see how to use this model to create a very simple web application using radio where you can simply upload an image and then that will give you the caption of that image as an output so that's what this video is about where you build an application where you can upload an image and you can get a caption outside so this model that you see right now it's called wit to digital gpt 2 because it uses width and also digital gpt 2 for extracting the text and then it has got the pretty much the code that you want to use i'm going to use this code almost it's not a code that i've built but i've just made a little changes in this code just to make it slightly cleaner and if you want to know more about how this code is created so there is a blog post associated with the model training process which i'll link in the youtube description you can check it out and also one thing that you need to keep in mind is whenever we talk about image captioning there is a very high chance that there are biases in the model that's something that we need to acknowledge something that we need to find ways to handle by the time i made this video i have done a couple of stress testing i've used certain images people of color people like different images people have like different gender images so i try to make sure that from the examples that i saw whether there is any bias in the text i could not find out but there is a possibility that you know you might see when you see this um you see this application deployed you might see some biases when you see buyers what can we do so we're going to also discuss about that so overall we're going to take this code use this model and create a web application but first i'm going to take you through this code line by line on a google collab notebook and then we are going to convert that into a radio application and then we are going to deploy that radio application as a web application on hugging face spaces so let's get started first thing is keep a google collab notebook ready so i've got a google collab notebook it's a cpu i'm not i'm not converting it to gpu i'm not running a gpu environment so with cpu environment i'm installing transformers hugging phase transformers quietly then i'm installing gradient after i do all these things now i'm importing image from pill below the library for image processing from transformers we are importing vision encoder decoder model with feature extractor and pre-trained tokenizer first so this is first to act as an encoder second this is for feature extraction and this is for the final text so the first two deals with image the last one deals with text for at least for the google column i wanted to take an image from a url and then use that image for the image captioning process so i'm using import request request library for http request to download an image okay so let me load all this and after we do this now the next step is we need to now download the model which is sachin slash bit to distill gpt2 so the same model that you see here sachin slash this this is what you're doing and let's download this model and after we do this now we need to download the feature extractor whatever you would like to call it you can call it feature extractor and this is directly coming from the wit base from google so directly google has already uploaded this model already and we are going to use that for feature extraction and then once we download that for feature extraction the next thing we need is we need to download digital gpt2 as a tokenizer for us so distill gpt 2 as a tokenizer so now we have all the models like all the liquid components in place we have the model we have the feature extractor and we have the tokenizer in place now let's specify the url from where we want to download the image so i'm going to use this so if you see this this is the image that i'm using to see what what does it show so this is the image with which we are going to first test it's not my image you can see the watermarks on it so this image let's define the url here and after we define the url now we're going to say to open the image get the image so this is where we are getting the image and we are opening the image and that image as image like you can see this with clause where we are calling that image and designing it image and that image is gone into this with feature extractor and then we are extracting the pixel values of it and after we do that now now we need to get the encoder outputs so the model that we have which is a vision encoder decoder model so we're going to encode the output and um that's what we're going to do now and save it in the object encoder outputs now after this is done now we are going to give this encoder outputs to this decoding like using tokenizer we're going to decode this and we are going to generate text that's that's basically what we are doing here so now after this one thing that i wanted to show is sometimes you get a repeated sentence as an output here so we want to do a very little text processing so you can see how the how the output looks like it says a woman riding a bike down a street that's what it says a woman riding a bike down a street and then there is a dot and then it repeats the sentence sometimes completely sometimes partially so what i'm going to do now is i'm going to do a very naive text processing post processing where i'm saying split this text with dot and then take only the first element it's a very knife text processing so i can say this is text processing or you can say naive takes processing let's run this and then you would see an output a woman riding a bike down the street so what do you see like when you see this image would you would you agree a woman riding a bike down the street i think it makes sense so now let's try to create this with a new image so to quickly recap what we have done we have installed the two libraries transformers and graduate software we have not used gradio uh and then we have imported the required components we have imported image we imported vision encoder model encoder decoder model with feature extractor pre-trained tokenizer fast and request library to collect the image and then we have load downloaded all the models so model feature extractor tokenizer and then we have downloaded the image given the image to the feature extractor then we in we use the model to create a encoder output and then use the encoder output to decode using the tokenizer and get the sentence that's it and we did a little bit of text pre-processing now let us quickly test whether this works fine so i'm going to go to google and then probably start looking for start looking for something where a lot of people walking okay let's see if it if it i think i'm going to be dead okay fine let's finally pick this image okay right click copy okay let me go to pixabay and then right click this open a new tab yes let's copy this url because you need to make sure that it ends with dot jpg otherwise you might face some problem or jpeg or png whatever right but you need to have the image extension i'm going to paste this in the url after i paste this in the url i'm going to repeat all the steps url feature extraction encoder output generate sentence and i don't have to show you this we can pre-process and see the output a couple of people standing next to each other in a parking lot so okay i didn't necessarily talk about the car but i think it does a decent job let me take one further image where i can say people people in a signal and see maybe i think a kid with a computer kid with a computer should be an image so i can go to google search for this image um okay so we have an image i can go copy the image address we have a jpg here so i can copy this come back to this google collab notebook and then i'm going to paste the url here paste the url here run run run run run run and see what's going to show it's still working out so downloaded the image i'm sorry download a young boy sitting on top of a laptop okay it does its job so it's it's not necessarily top of the laptop but but you know it it it works fine you can you can you can test with the different images so we can finally test with this one last image um and then we will move forward to understand how it work how to make this a web application with gradient oops i think i copied a wrong url um let's say copy let me click this link it should open this link from where i can download the image open image a new tab yeah we have a jpeg url and then just just like i said make sure that you have the image extension at the end don't run run run and then see what is it going to show so it's it's processing you can see step by step what's happening encoder outputs and redesigned and voice sitting on top of a laptop computer once again it says sitting on top but you can understand that it managed to connect these images and then it's it's trying to you know show you the output okay so for now we know that um we have not seen biases any like like i said right i tried with few images we didn't see biases and we know that you know it does its image captioning quite decent i wouldn't say like super awesome but quite decent i think it should be better than a lot of existing models i have not done any benchmarking but yeah you got the captions so now next step is to convert the script into a gradio web application because we want to deploy this as an application where people can upload an image and they can see the caption as an output one of the things that you need for gradius you don't you don't need to mention these expressions individually you need to create a function like an inference function for gradient okay so now we need to make sure how do we wrap this whatever we have created here as an inference function so what i'm doing here is i'm creating an inference function called bit to distill gpt2 that takes an image as an input okay that's what i'm doing that takes an image as an input and then it does three things extract pixel values extract encoder outputs extract generated text and then return the pre pro sorry post processed generated text so the last three steps that you saw here without downloading the image everything goes inside this inference function so let me call this inference function okay so now we have an inference function defined so the next thing is i'm going to call gradient import radio as gr and after that i have to specify the inputs what do i want to show i want to show a place where people can upload an image so that is my input gr dot inputs dot image i've got a separate gradient course that is on free code cam i would recommend you to check it out but if you do not want to know all the details i think this should be fine so you have got a separate input section where anybody can upload an image you've got an output section where you have got a text box that we're going to show as the image caption and then um this is this is uh we have to change the title so i took it from a different application so the title is image captioning using vit plus gpt2 okay so that is our title so the description the description is we can say vid and gpt2 are used to generate image caption for the uploaded image so we need to specify the model link so i'm going to use the model link here go back to our collab notebook we can say you know you can you can specify like an article here but to keep it simple we can just simply give to keep it simple we can just simply give the url and then say model repo on hugging face model hub okay and if you want to give more details you can give more details but i think this should be fine for us so we just have given a hyperlink for people to click and then go see the model and then you can specify some example images right now we okay so we need this example image in our current folder so i'm going to download this image called bike and then i'm going to give this bike image as an example image so this is an example image and then finally we are going to call the interface so we have input we have output we have title we have description we have article we have examples you can have multiple examples right now we are using only one example um but yeah if you want multiple examples as well we can do that and then we have the gradio interface function where we are specifying width to width to digital to gpt the inference function name and then inputs inputs outputs outputs title is equal to title description is equal to description article is equal to article examples is equal to examples and what is the theme that you want because we are going to deploy this on hugging face spaces i think it makes sense to have hugging face spaces theme and then we are going to launch it first to check it here before we deploy it on hugging face spaces so let's execute this and then see it says collab notebook detected you've got the url and then you're going to get the application here so let's see if it everything is fine it's loading so it says image captioning using vid plus jpt2 wit and gpt2 are used to generate image caption for the uploaded image and then you can upload the image here you have an example that you can select so i can select this example and then i can submit this image and then i can say let's see what happens three seconds right now we are not optimizing for inference there are a lot of improvements you can do right now i'm just deploying it so that you can play with this a group of people riding bikes on a city street i think it's fine it's it's a good explanation let me close this and then try to use a different image that i have on my local computer oops it doesn't work like that i cannot draw drag and drop it here okay i didn't know that um so let's let's upload it let's upload it on hugging face faces and then we can do so right now everything looks fine i think one thing that we didn't do is we didn't make a note about biases so i'm going to go edit edit this article whereas there's the article um and then say if you find any bias here or i can add it to the description as well then click flag okay so i want to say that so what we want to do now is we want to say we want to tell people in the description okay this image captioning model like many more models could have okay i think we need to say that it used coco dataset for training coco dataset was was used for training okay so then we can say this image captioning model might have some biases that we couldn't figure out during our stress testing so if you find any bias gender race and so on please use flag button to flag the image with bias okay i think this is quite important let's rerun this again to see if that text comes properly it's loading you can see so it says this image captioning model might have some biases that we couldn't figure figure out it's okay during our stress testing so if you find any bias please use the flag button to flag the image with bias so this is a flag button and i think we are um i think i think yeah it's very important for us to note it done okay so right now we have everything ready so what i'm going to do now is very simply i'm going to just comment these two okay so everything else looks fine but i don't need to download the model here oh sorry i don't i don't need any of these things okay so i'm going to comment all these because i'm going to convert this into a python script at this point i'm going to comment all these recommend this i'm going to comment this as well so i'm going to just keep this function ready where you can upload an image and we don't need this as well so i've commented all these things now i'm going to download go here click download downloader downloaded this file as an app.pi file as you can see the next thing is we have to go to hugging face spaces and then upload this model so go to google then say hugging face spaces and after you come here click hugging face spaces go to your profile um make sure that you are logged in i'm logged in in my case create new space after that give the new space name i'm going to say image captioning with with gpt2 okay what is the license i would like to give mit license streamlit oh no sorry it's a gradient application created public anybody can upload it so this is this is the basic file structure skeleton that you've got i can go here upload the file create the file so you can go to file version first i would like to give the requirements.ext file before i create anything require once i mean ideally this is not how you would want to create a requirements.txt file but because we were on collab environment i think this should be fine first thing we need is we need pi torch then we need uh transformers transformers is that correct yeah transformers then we need pillow um so top eye dots transformers below gradient is not required to mention here because um the login face by default gives you a gradient install so you don't have to bother about it so create the new file and then the next thing is go to file version add file upload file drag and drop this file the app.pi file that we downloaded from collab and then commit the changes just make sure one thing that by default you the hugging face spacers would expect your model to have the application to have app.pi file if you have a different name make sure that you give that name here and then again you have different emoticons like for example if you do not want to use this this emoticon you can go change it here i mean this is all the all the few changes that you can make to make it look better so right now you can see it's getting built you can go see the logs as well like what's happening if something goes wrong you can read it and then see one thing that i think we missed is that we might see a problem is we have to upload a bike.jpg image uh that we did not do so i i'm going to upload some image for from my local machine and then we can we can call that we can upload it there for example let's say people walking in the street okay i'm going to go upload an image from my local machine so what is it such a long name people walking in the street okay upload this and then copy this image sorry copy the name go to the code let while it is building you can actually change the code so that you know it doesn't show any error so in the examples we asked for bike image but here we are giving a different image yes that looks fine do you want to give more examples if you want to give more examples you need to upload more image let me let me upload an image maybe elon musk i don't know what is it going to say i hope it doesn't say anything controversial so upload an image get the elon musk image commit the image and save it and then find the image name elon musk.jpg jpeg you have to you have to remember the exact extension app.pi edit it and then you can go and then say of the second i have the second example which is what am i doing second example which is elon elon musk jpeg and then save it and then commit yep let's commit it and then see what's going to happen okay so it's getting built now so while it is getting built um i would like oh is it done already oh yeah i think it was it was done already but we were making the change um so it's getting built again so let's refresh this and we have to wait for the example the the first one is done um i think the second one might come let's see if it comes okay let's let's for now select the first one so click this image submit it and then see what happens a woman walking down a street with an umbrella she doesn't have an umbrella but maybe you know i just thought so you can see that it is finally loaded and you can see the images so quickly i made a mistake last time when you had seen so the example should be nested list so you need to make it a list not like you know um a string list so that's something that you need to keep in mind so we have the images ready so we can click elon musk and then submit it and then see what it says and um and you can say man is talking on a cell phone um not not very accurate so let's upload another image a man with the beard smiling so let me get another one and then see what happens here this is anikan siraj from their podcast i really liked it a man in a suit and thai holding the camera okay that doesn't look very accurate let's try with one final image before we stop our attempts a man with a beard is standing in front of a mirror he does a decent job so i've not i've not tried with images with a lot of components but um i think i feel you know it it works fine um but again like we said there could be biases um so just just make sure that you know um if you see any biases highlighted i have a feeling that you know we don't get to see logs here like when we see the biases so if you if you find any biases please please let the model um the author the creator of the model know and also you know you can let me know like so that we can we can see how to eliminate it within the code the application that we built so we can use some kind of stop words and then say okay like if these words are present in the output we don't want to show an output like it's always completely fine to not show anything than than showing something that is not correct so yeah that's it i have successfully managed to build an image captioning solution using um with digital gpt2 the model that has been uploaded by sachin once again thanks to sachin for uploading this model training and uploading this model and you can read more about it on sachin's blog post which i'll link it in the youtube description otherwise i hope this tutorial was helpful to you um i hope you are going to create something like this and let me know how it goes so thank you so much for listening to me stay safe happy coding