MiniGPT4: image understanding & open-source!

Key Takeaways

mini gpd4 is pretty cool it can describe images generate recipes from images and even make websites from an image it's just like the gpd4 demo that we have seen a month ago but gpd4 is not available yet mini gpt4 is completely open source you can play with it right now and it only took 10 hours to train this model so let's take a look together okay let's first of all take a look at the demo upload and start chat this step is taking a while okay so now we can start chat with mini GPT let's ask it to describe this image okay now I can see the description of this image which looks really really good this image shows that women with pink feathers on her face and a flamingo on her shoulder so now let's take a look at the mini gpt4 paper and see exactly how this model is made here is the underlying architecture of mini gbd4 we have an image here that feeds into a pre-trained model components of a visual model and then we have a single linear projection layer that project the visual features to the language model the language model here we have a kuna which is also open source the language model accepts the format of the prompt human the output of this single linear projection layer the question that we may ask here is a format assistant and then we get the output right here the logo design is simple and minimalistic it looks pretty simple it just combines pre-trained language model with pre-trained visual model plus a single linear projection layer on top of the visual model the visual encoder is blimp 2. with the vit and also the pre-trained Q former if you're interested you can take a look and then it uses the linear projection layer we just talked about there are two parts in their training process the first stage is the pre-training stage so in this stage they used a combined data set of the conceptual caption sbu and Leon 5 million image text pairs the display training stage only took about 10 hours to complete and used for a 120 gpus right so it's not a huge effort it's only 10 hours to train this model incredible right however there is an issue with this training step so they were struggling to produce coherent linguistic output such as generating words or sentences fragmented sentences or your irrelevant content so the stage 2 is fine-tuning before they fine-tune the model they need to have a better quality data set this is how they get this data set they use the model that they got from the first Prairie training stage so The Not So Perfect model to generate a comprehensive description of an image this is the prompt they used the image features which is the visual features produced by the linear projection layer describe the image in detail give us as many details as possible and the assistant will produce the description if it's less than 80 tokens and then a human will ask assistant to continue and we will just combine the outputs from those two steps and create a more comprehensive image description this way they were able to select 5000 images and generate corresponding language description for those 5000 images the next step is quite interesting they use chargpt to refine the description the use jhbt to fix the error remove any repeating sentences meaning as characters basically rewriting and refine the 5 5000 image descriptions in the end they were able to get 3 500 high quality image taxpayers satisfied with their requirements and now we can refine their prey trained model with those high quality image taxpayers and as a result mini gpd4 now is able to produce more natural and reliable responses the fine-tuning step only took seven minutes with a single a100 GPU wow so that's the architecture and two training stages I want to show you the data set that they end up having so I downloaded the data so we have all the images here this is image number two and we have this filter cap Json file which provides all the descriptions of the images and then you can see the description of the image here the image shows a man fishing on the long next to a river with a bridge in the background trees can be seen on the other side of the river and the sky is cloudy I don't see any sky but other than that everything is very accurate that's the data set and finally you can check out their page everything is open source and they have instructions on how to get started using their code yeah so that's mini gbt I just got excited when I saw this project and I thought I would share hope you find it helpful see you next time bye