Chat with your Image! BLIP-2 connects Q-Former w/ VISION-LANGUAGE models (ViT & T5 LLM)

Key Takeaways

hello Community today we combine our vision Transformer knowledge and our llm knowledge so we have vision and language and today I show you that there's a beautiful technique that we can now pre-train Vision language models based on the Transformer architecture and here we go this beauty is called blip2 bootstrapping language image free training with Frozen image encoders and llms this is exactly what we wanted interestingly here as you can see here the orders and it is from Salesforce research published January 30 2023 let's just jump right into it and you might ask but what is the use case for this technology so if I upload a thumbnail of one of my last videos let's have a look and we have here our chat input question what elements can you distinguish here on this picture and the answer is processing a helicopter a green screen and the words Vision Transformer now image to text task that visual language models can tackle our image captioning for the visually impaired great useful product description identify inappropriate content beyond the text then you have image text retrieval can be applied in multimodal search as well as autonomous driving and you have of course visual question answering what we are interested in and this will enable our multi-modal our jet GPT chatbots now multi-modal and yippee here we go so at first we have to make sure and this is now kind of a culmination what we learned to transform our architecture here on our large language models that I showed you uh T5 flan T5 Bloom everything here that you know either to decoder side or the encoder side and of course now Vision Transformers both have the same architecture this is the most important Point here both are transform architectures so we compare Apple to Apple and you're not going to believe it to bridge this modality Gap as they call it between Vision here on this side and language on this side they add now a Transformer Now isn't this a surprise so we have a Transformer that connects to a Transformer connects to another Transformer and we will know that our technique is absolutely compatible crossover so this Transformer we call a very Transformer you will understand on a second why this is or a q format so whenever you read now in February March 2023 starting that about Q former you know exactly what we're talking about the interface of vision Transformer to language transforms if you want to know more about llm I show you here about how you can use your flying T5 x6l or if you're interested to have your Bloom 176 billion parameter model operational on the AWS infrastructure those are my two videos for you if you want to learn in general about Transformer this is my video and one of my latest video on Vision Transformers are here on the left side so beautiful now we are of course interested the main problem is an llm is already a huge Model A Vision Transformer is also not as huge but coming up to 22 billion parameter my goodness these are monsters so if you want to combine them how do you want to train if this hardly fits in a supercomputer Center so this is not a beautiful idea here of the orders of this paper please have a look at the original paper that they say now we freeze the vision Transformer and refreeze completely all layers of an llm yes we have an interface between those we have an open i o channel to these models but the layers and the weights are frozen the only thing that we will train is between the Frozen Vision Transformer and the Frozen llm like chat GPT or whatever you.com chat or whatever gpd3 based methodologies you like to apply or flying T5 or whatever the only object we train here is our Hue Transformer a q format or querying Transformer this is the Hot Topic so how we do this now you're not going to believe it since on the one side we have a vision object and on the right side we have a language object the Q format itself has two modules it has an image object an image Transformer that interacts with the Frozen image encoder for the visual feature extraction and on the other side it has also a text Transformer that can function both as an encoder or a decoder for language so you see more or less we copy the external i o structure that it will attach itself on and we put some sub modules within our Q Transformer and yes you guessed it we will have common self attention layers within our Q format so if you want to have it in a little bit more detail this is the query for you you see here to initialize our queue form we use the weights from a bird base model you might say my goodness this simple bird-based model yes exactly now it all comes together we combine now the vision technology with language technology bird expert sentence Transformer this is it can you describe the elements in the image [Applause] now this is gonna be fun so let's have a look I've Fighter Jet flying in the sky with the words q a on jet GPT absolutely right you identified the fighter plane okay now which fight a plane 35 she not bad not bad the F-35 so how does the pre-training happen two stages in the first stage now the vision language representation learning stage as we call it we connect now our Q format that sits in the middle now with the Frozen image encoder so only with one side with our image encoder and perform the free training using some very specific image text pairs so you have an image and you have a text description of the contents of this image you have another image another content description in text of the content of this image and you get it you have then loss function and they found out the best way to do this is have three different loss function when you train it here on the first stage you have an image text contrasting last like we have and the normal birth systems also craned on then you have an image around the text generation and an image text matching loss all the details you find in their original research paper just to give you an idea we have here in let's call it yellow this something occurs monster year and we dock here our Q form in the very first pre-training step here to our image encoder that is frozen so image image encoder and here it is and now we have here as I told you two modules within our Q former and image Transformer and a language Transformer now you have now on the self attention layer you have sharing and this is more or less the main idea behind this if you want to have to text image transform extracts the fixed number of output features from the image encoder independent of the of the image resolution of course and receives learnable query embeddings as an input this here's our input I will show you in a second and then it runs true we have here our attention masking and be able to calculate our three different losses our three different methods and you have here beautiful described here how with the self-attention mechanism now working here across the modules now you might say okay in the Second Step well the second step you're not gonna believe it now we take the Q format it sits in the middle and connect it to the other part to the vision language generative learning where we're connected now to the Frozen NL llm in our case it will be a flan T5 XXL model now aquarium battings that we get out now have the relevant visual information to the text as it passes through an information bottleneck and these embeddings are not used as a visual prefix and I'll show you in a second what I mean to the input to the large language model and this pre-training phase effectively involves an image ground text generation task using causal language model loss let's have a visualization of this now they people that wrote the research paper decided to have two different models two different llms here for example you have here only only if you want a decoder based llm but you know if we're working with flying T5 we have a full Transformer stack so we have an encoder stack and a decoder stack so of course and we're gonna use the second option here for nt5 so we have here our input image that comes in our image encoder that is frozen then the first step is now done we have now the output here of our queue former and this here is runs now to a fully connected layer to linearly project the output query embedding Z into the same Dimension as now the text embedding for the llm and this is the beauty of course if you only have a decoder it goes directly if you have to feed an encoder and a decoder stack you have to take care about this a function here as a soft visual prompt and condition this on presentation extracted by the Q format yes the details you find in the paper I just want that you have an understanding a two process first one it goes in the Q format you got it you have a fully connected layer input to the llm and here now then the next word generation the generative AI takes place now the absolute beauty of this methodology here from the last day of January 2023 was that when they published blip 2 they use division Transformer and for the large language model as I showed you you can use the flan T5 model but you are not restricted to this pre-training approach since you freeze both Vision models and language models you both free stem you can combine almost any visual backbone with any large language model for this specific Vision language model development where you can train here this Q former so we get a complete pre-trained Vision language model out of the pipeline now you might say isn't that beautiful and I agree with you but you know what this was the theory so you understand what we're gonna code next time when we will code every single step we will build our own app maybe we even do agradio and we will end up with an operational Vision language Transformer model you can use you can apply so when you have an image you use the image as an input and then you can start to jet and the system automatically understands the content of the image and will respond to you in this chat correspondingly I say thank you I hope you enjoyed it a little bit see you in my next video