Flamingo paper - Comprehensive dissection

Key Takeaways

Hi, welcome back to a new video from reading research papers series. Today we'll be picking up Flamingo paper. So I have the copy of paper printed here. Flamingo. Uh this paper is available on archive. I will show you the link. But this paper was published by Google deep mind and it's very very popular. It's presenting a vision language model with few short learning capability. We'll briefly discuss what is a vision language model, what exactly is few short learning etc. Uh there are large number of authors here. If you see this uh section of the paper, the these are all author names. There are there are so many authors in this paper and the length of the paper counting the main section plus appendix. It's 54 pages. So to not prolong the video so much. I will not be going through the appendix part that much. I'll go through the main paper and I'll go through some of the figures in the appendix that are relevant for your understanding of the flamingo model. We will also discuss why this model was named as flamingo. What is so special about this model? how popular is this in the context of uh multimodel LLMs etc. So with that uh let's get into the paper dissection. I'm hoping to do this dissection um in just one sitting of I don't know 1 and a half to two hours or something like that and after I trim some of the parts of the video hopefully this won't be a too long of a video but I also don't know right I'm also uh recording while reading the paper for the first time although I have read the paper a couple of times before um and implementation of flamingo model is also not that straightforward it's a little bit um bigger model to train from scratch and the the weights of flamingo train model is not available in public. However, there is a model called open flamingo which was introduced as almost exact replica of the deep minds flamingo model but in an open-source manner. We'll also go through that GitHub repo really quickly. We will not be going through the code but we'll be just be opening the repo and seeing what they have done if we were to run inference what exactly can we do etc. So with that let's get into the video. So here on my whiteboard I have pasted the individual pages of the paper separately. So we can start uh start now. So you can just Google flamingo paper. Uh it will show up. It has around 6,700 citations. You can see that here. And paper is uh available on archive for free. So you can simply go ahead and download. If you want to print it and read um through a printed version of the paper by annotating certain things feel free to do that. So that'll be a good exercise for you. But the the title of the paper is flamingo a vision language model for few short learning vision language model or VLM is a multimodel large language model. So it can not just have text capability it can also process images right. So we will see that there will be a image encoder and also um an encoder which can handle text. So there are large number of authors in the paper. So um so I won't be attempting to you know pronounce their names or mention their names because there are so many people who contributed to this paper but we'll go ahead and read the abstract and the paper is from Google deep mind. So abstract building models that can be rapidly adapted to novel tasks using only a handful of annotated examples is an open challenge for multimodel uh machine learning research. So usually we know that if it's a classification task or any sort of deep learning related task you need a lot of examples for supervised learning if it's a classification task even for binary classification a lot of examples are typically provided. So this is a you know a challenge uh because in multimodality if you need annotated images annotated text etc or some kind of data set where where there is a uh you know input label pair it's very difficult to get in multimodel models. So we introduce Flamingo a family of VLMs with this ability. When when you say this ability here, this refers to um ability to learn just from a few annotated examples. Uh let me just change the size of my uh marker. So by only providing a handful of examples which are annotated, can we have a vision language model with lot of capabilities? So this paper introduces a few different uh innovations in uh specifically three innovations which are mentioned in the abstract itself but which are expanded in the main paper. So first is you utilize an already existing large language model and already existing um image model. So there are already lot of pre-trained uh vision models and pre-trained language models. So why can't we use the capabilities of those and somehow bridge it right? So that is the first innovation bridge powerful pre-trained vision only and language only models. So this is the first thing. Second innovation is handle sequences of arbitrary arbitrarily interled visual and textual data. What this means is um you might have seen certain documents where you have some text followed by some image then again there is text uh followed by image. If you are writing a document in micros uh in Google doc or Microsoft word you can actually copy paste images within the text. It can form part of the text right. So that is called interle um visual and textual data. So if you have a sequence where a sequence consists of words then maybe some image then again words followed by some image something like that. If there are sequences of arbitrary length or arbitrary order in which uh you have uh text and image embedded you this model is able to handle that then seamlessly ingest images or videos as input. So not just images it should be I mean this flamingo model is also able to take uh images as a input. So we will see how the model is constructed and how it has these capabilities. One question you may have is this flamingo is actually the name of u this bird. So this is flamingo actually um this bird. So I had a doubt when I was reading the paper first uh why is the model named flamingo you will see that soon and of course we'll be using chat GPT as much as we need uh to answer our questions and to make this process of reading the paper much more seamless. So there is no you know if you are reading a paper you you must be comfortable um using charg or even if you are taking a lecture um if there is a doubt you have one of the most powerful intelligent uh tools with you so why not use it right so we'll be using charg to answer some of our questions while we while we read the paper flamingo models can be trained on large scale multimodel web corpora so this data set will consist of text and image this multimodel web corpora. We'll see what that data set looks like containing arbitrarily interled text and images which is key to endo them with a in context uh with in context few short learning capabilities. So now there is a question what exactly is few short learning right um let's ask JGBT what is f short learning but in short typically few short learning is with a few examples given u the model should be able to make a prediction so in figure one of the paper itself there is a very nice example uh given actually it's over here there is a very nice example given here this is a chinchilla they are mainly found in Chile so this image is given then this text is given then this image is given this text is given this is a shea These are very they are very popular in Japan. And then an actual image of flamingo is given. Then the text is this is then the model should produce text as the output. So please note one thing flamingo model does is it does not produce image output. It's a generative model which can generate text as the output but but the input can contain images as well as text. Uh so it is a multimodel LLM not from the output perspective from input perspective. It can input process multimodel multi multiple modalities but output is just text. So you might uh remember clip model. Clip model is um using contrastive learning. You could train uh a vision language model to understand images as well as text. But in clip model all that we were trying to do was um let's say there is an image of a cat and a text that that writes cat. Both of these things can be represented using vectors. And in vision language model, the way in which image and text is aligned is by ensuring that these vectors which represent same idea. So if it's an idea of image of a cat or text that writes cat those two vectors um should have very high cosign similarity. So the angle between those two vectors should be uh very small. So if that can be maintained, so if this angle can be very small, uh it means you have vision language alignment. But one problem with click model was that you could not you know um generate anything. You could only create alignment between vectors. You could not produce an output as a result a new output. It could say if you give an image from the captions that this model was trained on which caption might be the best fit for the image or vice versa if you give a caption which image might be the best fit for this caption that the model can do but it could not generate anything but flamingo model can generate. So that is one uh great thing about this model. But let's ask chat GPT what formally what what it can say about um you know few short learning. What is few short learning? Explain in two sentences two to three sentences. I don't need a long reply and I don't need thinking. I just need instant uh instant model. ML approach where model learns to perform a task using very small labelled examples of often one to 10 examples per class instead of learning from scratch. The model relies on prior knowledge learned during pre-training and quickly adapts to new tasks with minimal data. Right? So the the term few short learning is almost self-explanatory. All right. So now let's continue with the uh remaining portion of the abstract. Um, we perform a thorough evaluation of our models, exploring and measuring their ability to rapidly adapt to a variety of image and video tasks. These include open-ended tasks such as VQA. VQA is visual question answering where the model is prompted with a question which which it has to answer. So if there is um an image given and a question associated with the image which is also given in this figure example where is so this one is this one is like that right? This is asking here is the image what is this? But to for the model to be able to predict it's given couple of examples here. So this is called few short learning. Here only two model is given only two examples. Here is another one. So here model is given an example. Um if the proper example was not given model might not realize that it has to do a mathematical calculation. It might just say what is this? So so if the model was answering what is this? This is a flamingo. This is some what is this? Some city uh painting. This is some uh board. This is some mathematical operation. Right? But that is not what the model is supposed to output in every single example. So here in this example we are explicitly telling the model for this image what the model is supposed to do is do 2 + 1= 3. Here the model is supposed to do 5 + 6= 11. Similarly here the output should be 3 * 6 equal to 18. And the model will output that. So um just see the nature of the output. It's it's not just describing what the image is. Sometimes you want to describe the image. Sometimes you want to use the information from the image to calculate something else. Right? Uh here just see this. So here the board output is underground. Here the board output congress. So here the output is uh what is this? Solom. So these are these are examples of different examples of short learning. So the model in general understands text and understands image. But depending on whether the task is visual question answering or or some you know mathematical calculation to be done based on some uh input or something else model can do that or classification if model has to classify an image into cats versus dogs. If we give a couple of examples it can do that. Um so wiki where model is prompted with a question which it has to answer. Captioning task. So you give an image and ask the model to put caption which evaluate the ability to describe a scene or an event and close-ended task such as multiplechoice VQA. Multiple choice VQA is almost like multiclass classification. You have visual from a given set of questions. It has to answer one thing. Uh for tasks lying anywhere on this spectrum. What is the spectrum here? spectrum of task including VQA, visual question answering or uh captioning or multiplechoice VQA for all these all tasks ranging in this spectrum a single flamingo model can achieve new state-of-the-art with few short learning simply by prompting the model with task specific examples we just saw that on numeric on numerous benchmarks flamingo outperforms models fine-tuned on thousands of times more spec more task specific data which is amazing right because flamingo is not a general purpose model but it can be you know with few short learning ability it can be tuned to perform good in any almost any kind of tasks in the spectrum of these three tasks and this paper was uh accepted to new uh 2022 which is a very very big deal all right so now let's move on to the next page abstract is done so next page is basically this full figure uh we can just walk through the example so here the example is um animals what are these animals is through two examples here. Look at this. What is the title of the paint this painting? Answer the hallucinog hallucinogenic uh toodor. What is the this painting? Where is this painting displayed? Answer lur museum Paris. Then another painting is given. What's the name of the city where this was paint? and then answer is. Um I am not familiar with these I mean this painting but uh in any case the model is given some examples and um based on the example model learns what to focus on uh when a when a new question is asked and then it can answer then let's take another one. Yeah. So here the model is given an image in which there are three pandas right and then uh expected output in this example is pandas 3. Another example that the model is provided is this image dogs 2 and here model is just provided an image but you know there there are like what four giraffes right so giraffes 4 is the output that model gives so this whole thing is the input okay just please understand this whole thing is the input and within this whole input the model is given a few two examples with two examples for a new unseen data point model can make a prediction and um here is Another example where you the model can take uh video frames as the input. So this is actually different frames of a video and then model is asked based on the video what happens to the man after hitting the ball and answer is you can see here that the man is falling down. He falls down. It's probably difficult to you know analyze that from just this frame. Uh because in this frame you don't know whether the person is falling down or whether the person is crouching or just just kneeling down on the ground. Who knows? Um but with these entire frames you can make out that the person is falling. Same thing the model can also uh understand. Then this example is that of multiple uh uh you know image visual dialogue. So here an image is given. This is a picture of two teddy bears on moon. So this is what the model is giving as the output. What are they doing then? They are having a conversation. What object are they using? It looks like a computer. So basically for a given set of images or for a given image you can have multiple uh sort of like Q&As possible with the flamingo model. One question is why is this model named as flamingo? We can just ask JGPT. My thinking is they just I mean they don't explain it I think anywhere in the paper. Why is flamingo model named? So I think this is based on that first example which you saw chinchilla and um um what was it? Shea and last third example was flamingo. So maybe that was the reason flamingos are known for their ability to stand on one leg. uh few short learning modalities. Uh it balances vision and language. Okay. Okay. So, uh flamingos have very good balance on one leg. Um here vision and language it can balance. Okay. I don't think I don't think this name has too much significance. I I really don't know what exact you know metaphorical reason why authors picked the name flamingo but whatever we can I think it's it might be explained somewhere in the appendix we can we can uh do a quick search um so this is the first uh I mean first figure uh in the second page flamingo can rapidly adapt to various image/v video understanding tasks with a few short prompting out of the box flamingo is also capable of multi the image visual dialogue. So these are some examples more more examples are available in the appendix. Then figure two we can read in a little bit after we read the introduction. Uh in any case figure 2 is showing um the results from training the flamingo on different tasks. But before going into figure two let's read the introduction section. Introduction. One key aspect of intelligence is the ability to quickly learn to perform a new task given uh let me just move a little bit here. Perform a new task given a short instruction. So this is a key aspect of intelligence. I agree. While initial progress has been made towards a similar capability in computer vision, the most widely used paradigm still consists of first pre-training on a large amount of supervised data before finetuning on the finetuning the model on the task of interest. So uh here when they say initial progress has been made towards few short learning in computer vision, I am not 100% sure which model they are referring to. So maybe we can ask JGBT Here which model are they referring to when they say few short learning was attempted in computer vision. not uh referring to specific model but these are some models. All right. Okay. They are not referring to any specific model. Um the most widely used paradigm still consists of first pre-training a large amount of supervised data. Uh pre-training on a large amount of supervised data before fine-tuning the model on the task of interest. All right. uh which means you have to do a lot of um hard work in pre-training before finetuning for the specific task which means it's a double amount of work. What if you have to fine-tune for six different type of tasks? You have to do finetuning six times. However, successful finetuning often requires many thousands of annotated data points. In addition, it often requires careful per task hyperparameter tuning. This we have discussed in many papers. Finetuning is actually not easy. Although the amount of changes happening to the model is less in finetuning compared to pre-training, it's still expensive. It's resource intensive. Recently multi multimodel VLMs trained with contrastive objective. So here they will be referring to clip paper primarily. We can check 50th and 85th one of these must be the clip paper have enabled zeros short adaptation to novel tasks without the need for finetuning. However, because these models simply provide a similarity score. So this is what I was explaining earlier. It simply provides a cosign similarity score between text and image. All right. They can only address limited use cases such as classification where a finite set of outcomes is provided. So classification or um image captioning where caption is not generated. Caption is picked from existing set of captions or uh image identification for a given caption. basically image searching searching to find the right kind of image for your search query. It's basically considering query as a caption and then finding the image that best fits the caption from your data set. They crucially lack the ability to generate language. So the modules like clip they cannot generate language which makes them less suitable for uh to more open-ended tasks such as captioning or VQA. Uh so because captioning has to be truly generative. If an image is shown, you are generating a new caption. It should not be like because you have 400 million examples in your um training data. You can pick one of the captions from your training data and say that hey that caption is the best fit for your current image. That is what clip model does. But that's not the best thing, right? It's not generating caption. Others have explored visually conditioned language generation but not have not yet shown good performance in low data regimes. So that is what this paper is uh showing. It will show this what this visual conditioning of language means. Uh we will see that when we proceed uh in the rest of the paper. We introduce Flamingo, a VLM that sets a new state-of-the-art in few short learning on a wide range of open-ended vision and language tasks simply by being prompted with a few input or output examples. So you already saw this in figure one. Of the 16 tasks we consider, Flamingo also surpasses the fine-tuned state-of-the-art on six tasks despite using orders of magnitude less task specific training data. So this is where figure 2 comes into picture. We will look at figure 2 in a moment. To achieve this, Flamingo takes inspiration from recent work on LLMs which are good few short learners because you might have seen LLM which is like CHP for example. It is a it is a language model trained to predict the next word. But it can do a lot of things. Uh it can do mathematics. It can do translation. It can write poems. It can um you know analyze a research paper if you copy paste it. A lot of things it can do and all of those properties are uh can be you can bring out those properties with zero short meaning you don't even have to give an example. You just need to ask it what to do or few short in some cases. If you want to give an example of what the model is supposed to do, you can give one or two examples in the prompt and the model can do it directly. Right? So this is why it's a it's a good few short learner. LLM large language models are in general good few short learners. Meaning you it does not need large number of annotated examples for it to perform really good in new tasks. A single large language model can achieve strong performance on many tasks using only its text interface. A few examples of the a few examples of a task are provided to the model as a prompt along with query input and the model generates a continuation to produce a predicted output for that query. So corresponding to query output is produced but along with the query in the prompt you are giving some examples. We show that the same can be done for image and video understanding tasks such as classification, captioning or we or question answering. These can be cast as X prediction problems with visual input conditioning. So what is classification task? Classification task is essentially you give an image and say hey can you classify this into category cat or dog. So you provide the image and you provide a text sentence and then it has to produce a sentence output. Captioning is what captioning is you provide an image and you say hey can you write a good caption for this image and then it has to produce a text output. In visual question answering you provide an image and then then you ask a question in this image um how many animals can you count and then it has to produce a text output. So in all these cases you have visual input conditioning uh input contains text and image and output is basically uh a text. The difference from a language model is that the model must be able to ingest multimodel prompt containing images and/or videos interled with text. So in vision language model or if we are building flamingo it should be able to text take text as well as English. Flamingo models have this capability they are visually conditioned auto reggressive text generation models. What is auto reggressive text generation model. See GPT is an auto uh regressive model. Whatever output it produces as the next token it takes it adds to the existing set of tokens and then the next word is predicted. So I word is predicted then that is added to the existing um u you know set of tokens then I + 1 is predicted then that is added to the existing set of tokens like that it it is auto reggressive meaning its own output is used to produce its next output. So here this flamingo model is also visually conditioned auto reggressive text generation model. It is a text generation model not an image generation model but its input can have image also. So it's a visually conditioned auto reggressive text generation model. Why auto reggressive? Because the the text it produces is added in to its input to produce the next token and then it's repeated to produce the next token. Um it's able to ingest a sequence of text tokens interle with images or videos and produce text as output. Flaming flamingo models leverage two complimentary pre-trained and frozen models. So this is the most important thing you should know. Flamingo does not train uh its vision encoder or language encoder completely from scratch. It use pre-trained models. A vision model which can perceive visual scenes and a large language model which can perform a basic form of reasoning. Novel architecture components are added in between these models in between these two models to connect them in a way that preserves the knowledge they have accumulated during computationally intensive pre-training. So pre-training of language model or vision model is very expensive. So whatever the model has learned during pre-training should somehow be preserved during the process of converting it into flamingo model. So they have done a few things to make sure that that happens. Flamingo models are also able to ingest high resolution images or videos thanks to a perceiver based architecture. So we we are yet to see what is this perceiver. uh we'll see that in next figure I think that can produce a small number of fixed uh a small fixed number of visual tokens per image or video given a large number of uh large and variable number of visual input features. So let's say you input a 40 secondond video versus a 1 second video or video resolution is different. the the input has variable size but the immediate output which is used for you know creating let's say sort of cross attention between language and uh image has to be of fixed length. So how does it do that? A perceiver based architecture is used for that. We will see what it looks like. A crucial aspect for the performance of large language models is that they are trained on large amount of text data. This training provides general purpose generation capabilities that allows these language models to perform well when prompted with task task specific examples. Similarly, we can demonstrate that the way we train flamingo is crucial for it for its final performance. They are trained on a carefully chosen mixture of complimentary large scale multimodel data coming only from the web. So flamingo model is trained on complimentary large scale multimodel data. Let's not focus on the word complimentary for the moment. It's large scale meaning a lot of data points are there for training flamingo. Uh but then if we are training flamingo in such a large data set then why do we use pre-trained model? Because pre-trained models already know a lot of things about images and lot of things about text and uh whatever changes we are making to the pre-trained vision and pre-trained language so that it can take both image and text as the input that is flamingo model and then additionally flamingo model is again trained on multimodel data because originally vision model is only trained on vision language model is only trained on language no multimodality is there so once flamingo architecture is constructed although these individual models are pre-trained The whole thing has to be once more trained on multimodel data and this data is coming from web without using any data annotated for machine learning purpose. So there is no human annotation of data in this at all. After this training a flamingo model can be directly adapted to vision tasks via simple few short learning without any task specific fine-tuning. This is what it is what is amazing about the model. So although the paper has not referred to figure 3 yet I think we can quickly go through the architecture of figure 3. Then when they describe the figure we will understand it better. Then once more we can come back to figure three. So let's just focus on figure three for a moment. So this is flamingo architecture. So let's start from this gray rectangle. So this gray rectangle is the input. In the input first there is an image then there is text. Then there is image then there is text. So this is there is a dog. The image of a dog and then the text is this is a very cute dog. Then the image is this is so here can you try to predict what the output might be? uh like just you as a human what would you say here? This is not just saying this is a dog it says this is a very cute dog. So there is an adjective associated with dog right. So based on that logic if you are a human and if you are you know extending this you will say something like this is a very angry cat or something because the cat's face look a bit angry right so the output this model is expected to produce because please remember flamingo model only produces uh uh text output it does not produce image so the output you are expecting from this is something like this is an angry uh cat so now let's see what happens so first uh this whole thing has to be converted ed into text only part and image only part because image encoder can only take image. Text encoder can only uh take text. So before passing this into text encoder what is done is wherever there is image an image token something that looks like this is added. Okay. Remaining is text. So text text is preserved as such. So after processing this interled visual text data the process text will look something like this. So whenever there is an image it's replaced with an image tag. So image this is a very cute top image. This is so here in this text there is no information about the image. I mean this image tag is common irrespective of what the image is. So this image tag and this image tag are not different. They are the same thing. Uh and then this process text is passed into a language model. So this is language model with some modifications. So I will talk about the modifications in a moment. Then what happens to this uh image? These images are passed. There are two images. So it's passed into vision encoder. So the dog image goes through a vision encoder. Cat image goes through vision encoder. It's the same vision encoder. And uh vision encoder will will encode it into certain token length uh taking in all the context. So this vision encoder is a vision transformer model which the paper is yet to explain but we are still going through the figure. And then this perceiver resampler has a very specific task of you know irrespective of the size of the input uh this perceiver uh resampler will produce output of same length. So then now whatever is coming here are context vectors associated with image. Whatever is coming here is context vector associated with text. Now you will see this symbol this uh you know ice uh snow symbol. It means this is a frozen layer. This is frozen meaning this vision transformer is frozen. So pre-trained weights are not changed. But this is a learnable layer. This is learnable. This is learnable. This is learnable. But this is frozen and this is frozen. You see this uh you know snow snowflake symbol. So what they have done is they have taken a language model in which all the layers of the language model with self attention plus multi-layer perceptron they have retrained as such. But in addition they have added some layers extra layers. These these are extra layers called gated cross attention. So X attention is cross attention. So uh cross attention dense. So we will go into the details of if we expand this layer how does it look like? Obviously it will have a cross attention. It will have u residuals. It will have multi-layer perceptron. Here it's almost the same. It will have of course uh self attention. It will have uh multi-layer perceptron and it will have residuals. And layer normalization will also be there in both of these. But uh when we draw layer normalization is typically not drawn to keep the diagram concise. But there is an expanded version of the same diagram in the paper which we will go through in a little bit. But basically this is what is done. Okay. So here the tokens corresponding to image is coming. Here the tokens corresponding to text is coming and cross attention is created between those two in every single layer. Self attention is created for the entire in the uh corresponding to entire token sequence in this language model block. And finally the output is a very serious cat. So see the output is not just this is a cat. It says the output is a very serious cat. Uh here if the in the example if the input was like this is a dog. Let's say this part was not there. Very cute was not there. Here the output will be simply this is a cat. But here since this very cute adjective is given the model understood that you have to also add an adjective for the cat. So it says this is a very serious cat. Very nice right? Very good example they have selected. that the authors have selected. Now let's move into uh contributions section. Contributions. In summary, our contributions are the following. We introduce the Flamingo family of VLMs which can perform various multimodel tasks such as captioning, visual dialogue or visual question answering from only a few input output examples. Thanks to architectural innovations, the Flamingo models can efficiently accept arbitrarily interled visual data and text as input and generate text in an open-ended manner. So the interle of text and image which is shown here, it can be in any fashion. You don't you don't have any token constraint saying hey you need this many text tokens before you can have an image token. There is nothing like that. Then that is the first thing. So interle of textual and image data to produce uh textual output. So input is text and image output is um text. Then we quantitatively evaluate how flamingo models can be adopted to various tasks via few short learning. We notably reserve a large set of held out benchmarks which have not been used for validation of any design decisions or hyperparameters of the approach. We use these to estimate unbiased fshot performance. So here they are talking about the benchmarks that that they have used uh to evaluate the flamingo model. Uh flamingo sets a new state-of-the-art on few in few short learning on a wide array of 16 multimodel language and image image or video um understanding tasks. On six out of these 16 tasks, Flamingo outperforms this fine-tuned state-of-the-art despite using only 32 star specific examples. So in figure one they showed only two examples per task but here to to uh overcome the state of the uh the the state-of-the-art before flamingo they used 32 examples around thousand times less uh task specific training data than the current state-of-the-art. So the current state-of-the-art had probably 32,000 uh on that order of magnitude amount of data for finetuning with a larger annotation budget. Flamingo can also be effectively fine-tuned to set a new state-of-the-art on five additional challenging benchmark. VQA, V2, VAX, vis, MSR, VTT, QA, and hateful meme. So, we'll see what these benchmarks are. There are many many benchmarks by the way. So, figure two we have not gone through yet, right? Uh it was showing this graph with the performance. Maybe we can read through this a little bit. Flamingo results overview. So this u you know uh what is this maroon color is flamingo 80 billion model with 32 shots and these gray color ones are previous zero or few short soda. Sot means state-of-the-art. This these are state-of-the-art. So here it's a performance relative to uh few short f soda is what I think I forgot what was this ft sort maybe it is few short training state-of-the-art. What does FT soda mean? Not f short. Yeah, of course. Yes. So, finetuned state-of-the-art. So, FT sort of means fine-tuned state-of-the-art. So, performance relative to fine-tuned state-of-the-art. Um, so yeah. So, finetuning is very expensive. So, you are comparing few short learning against fine-tuned models. In few short learning uh itself there are there was the previous state-of-the-art and there is flamingo and here you can see that in six task so if state-of-the-art uh I mean if finetuning performance is 100%age because you are normalizing the few short performance relative to finetuning performance and here you can see that in 1 2 3 4 5 6 out of these 16 tasks uh performance of um few short from flamingo is better than performance from um finetuning and also for all these previous uh state-of-the-art for few short learning the flamingo model is performing better. So this relative to finetuning this is only performing 41%age flamingo is performing 66 here it is only 15% flamingo is 75 and so on. So in none of these examples no other previous state-of-the-art for few short learning is overcoming flamingo and similarly in 16 of these uh examples. So this is flamingo 80 billion model with only 32 training examples 32 you know few short examples whereas this is fully fine-tuned uh you are not you are comparing this 32 example based finetuning of flamingo with um fine-tuning. So if a number is less than 100%age it means the flamingo model with few short is performing worse than finetuning. If the number is greater than 100 like this 106 109 115 and so on. It means flamingo model is performing better than uh finetuning with just 32 examples. So here in six tasks flamingo is performing better. Okay. And here it's showing the performance of different models. So of course um flamingo 80 billion being the biggest model it has the best performance. So that uh makes sense. So let's read this. Left our largest model dubbed flamingo outperforms state-of-the-art fine-tuned model on six out of 16 tasks. Uh we consider with no finetuning for the task for the nine tasks with published few short results. Flamingo sets new fine few short state-of-the-art. So here actually in the bar plot there are only 15 bar plot because in the 16th task I think um the you know uh fine-tuning state-of-the-art is not available. So our 16th benchmark which is a rare act is omitted because it is a zeros benchmark with no available fine-tuned result which means you cannot compare it with against 100%age because you don't know what the fine-tuned result for that particular task is. So that is not plotted here. But in the in all the tasks, Flamingo sets new state-of-the-art for find few short learning for sure. And in six out of 16 task or rather six out of 15 tasks, Flamingo sets state-of-the-art even in comparison against um finetuning. So that is what is done here. Okay. So figure two is pretty straightforward. Now let's move into next section approach. This section describes Flamingo, a VLM that accepts text interled with images for videos as input and outputs free from text. Free form text. The key architectural components are shown in figure three, which I think we looked at right. Yeah, this is figure three um are uh in figure three are chosen to leverage pre-trained vision and pre-trained language model and bridge them effectively. First the perceiver resampler receives spatial temporal features from vision encoder. So spatial temporal features why do why does it say temporal isn't image just spatial? Where is temporal coming here? Temporal is coming because if the input is a video it will have frames. So it will have temporal information also. So spatial temporal information coming from the vision encoder is passed into um perceiver resampler. um and uh so it receives spatial temporal features from vision encoder obtained from either an image or video and outputs a fixed number of visual tokens. So perceive resampler irrespective of the number of input it outputs a fixed number of visual tokens. Second these visual tokens are used to condition the frozen language model using freshly initialized cross attention layer. So this is why this cross attention is uh written as X attention cross uh in the um in this purple color this gated X attention dense. So gated it's there is a tan hyperbolic gating. So I will show you why this gating is needed and what exactly it looks like. It's not complex at all. So don't don't worry by seeing the term. Then um these new layers offer an expressive way for language model to incorporate visual information. So see here these new layers take in information from the image. So these image information is coming into the new layer. So this new layer is which one? This purple color layer. The blue color layer is already existing layer in the language model. It's pre-trained. It's frozen. Okay. So new layer helps language model to also take image context and produce cross attention result so that it can produce even better context vectors. So uh incorporate visual information for the next token prediction task. Flamingo models like flamingo models the uh flamingo models the likelihood of text y conditions conditioned on interled images and videos uh x as follows. So p of y given x. So y is the predicted uh next token, x is the input and uh basically the loss is similar to uh a cross entropy loss. Basically there are set of tokens in the dictionary and you are trying to predict which token among the dictionary is your next token. There is one token which the model is supposed to predict because um if you have an input text and if you are using the model to predict the next token there is one token that the model is supposed to predict there and there is one token that the model actually predicts. So based on actual prediction there will be a softmax probability distribution um and uh but there is there is a token that it is supposed to predict. So using the softmax probability distribution versus the ground truth you can construct a loss function using cross entropy loss. So where y l is the language token of the input text and y less than l is the set of preceding tokens. X less than or equal to l is the set of images or videos preceding the token y l in the interlude sequence and p is the parameterized p is parameterized by a flamingo model. So this equation is uh just a mathematical representation of what we already understand from this image. So here if it is predicting some elf token, it has all these text tokens preceding it and these two image tokens preceding it. So these image tokens plus this token converted into this kind of a format will be used to predict the next set of tokens. And here what are those next tokens? This one a very serious cat. Um the ability to handle interled text and visual sequences makes it natural to use Flamingo model for in context few short learning analogous to GPT3 with a few short text prompting. The model is trained on a diverse mixture of data sets as described in section section uh 2.4. So now we can move on to the next page. The next page starts with figure 4. Um although figure 4 is not yet referenced in the paper we can read through it uh quickly. So figure four is basically the expansion of this individual uh layer. So this is uh block number one. This is then there is block number two like that there are n blocks. So nth language model block nth gated uh cross attention block. So this is one such blocks. Okay. So in one such blocks input this x is what? Input coming from image. This y is input coming from text. Now in the gated cross attention uh there is cross attention between text input and im uh image input and text input. Then in language model layer which is frozen, it converts uh you know these inputs into another set of context vectors like this. There are n number of uh layers. Now this is an expansion of one such layer. So this expands to we already know right in a language model transformer block there are two things self attention followed by multi-re perceptron or a feed forward neural network. So that is feed for this is FFW feed forward uh neural network or MLP and before that there is self attention similarly in cross attention what will what will be there there will be one multi-layer perceptron and before that there will be cross attention module so now let's see what this cross attention means. So usually in self attention the queries and keys are both produced from the same sequence right same sequence uh a sequence is attending to itself but in cross attention quiry will come from different u uh sequence and key and value will come from another set of sequence. So let's start with language model layer. So because this is something we are more familiar with right. So whatever is coming in is converted or projected into query key value space. So this whatever is coming in is y. This y is projected into uh quiry. It's projected into key and value. So key equal to value equal to function of y. So here it does not say that key and value are the same. It just simply says same sequence is used for producing key and value. Same sequence is produced for used for producing query. And then query key and value will be used for producing uh softmax of query.pose divided by square root of d. Right? So it's basically the self attention formula. And then there are residual or skip connections here. This is uh one residual connection will go around self attention. One residual will go around feed forward neural network. Feed forward neural network. What does it do? It has one hidden layer which projects your whatever is the dimensionality of your uh context vectors into a 4x higher 4x dimensional space. So if the context vectors had 768 dimension, it'll be projected into 4 * 768 that many dimensional space and then it'll project back. Now this part is simple. This if you know the transformer architecture, this part is not at all uh complex. Now what about gated cross attention in this query is coming from language. Key and value are coming from vision. And where is this vision input coming from? It's coming from the vision encoder and perceiver resampler. So from that vision input is coming which is projected into key and value space. And what about the input for the language? It's the sequence of language tokens where image is actually replaced by an image token. Right? So this language input is projected into query. Um then this vision input is projected into key and value and then cross attention is constructed between them. Now here there is a tanage gating. So what the tanage gating does is look at where the tanage gating is getting applied. is it's basically getting applied to the context vectors created using uh this language and vision output. So what what this was try what this is trying to do which we will read soon is to make sure that the l the the vision input is not overpowering this language large language model module because at the end of the day this whole thing is primarily a pre-trained language model with some extra layer added in between. So this is pre-trained part with extra layer. Uh so all these dark blue ones are pre-trained frozen layers. All these purple ones are uh newly added layers. So primarily this is a language model. So if too much vision input is injected into it, it may not be good. Uh we'll read reasons why that is the case. But basically this tannanish gating will ensure or it will give you a way to control the amount of information uh vision related information uh flowing inside this uh language module. Okay. So tan is gating followed by residual connection. So residual connection again there is one residual around um you know uh this one around the cross attention one residual around the feed forward uh neural network. Feed forward neural network also has tag gating and here cross attention uh the residual around the cross attention is residual of the query itself. So query is directly passed here. Meaning you can see that language input has more significance here. Why? Because here language input is directly passed here after the tan gating right. If language also had to be suppressed or controlled a little bit through tan function uh it would have passed through this. So and also what is tan? Tanish will have a graph that looks like this which has minus1 to 1 as the range. So tanish makes sure that no matter what the magnitude of the input the magnitude of the output will be between minus1 and one. So it's a bounded uh function right hyperbolic. So uh that is what is done here but more details we will read in a little bit but let's read the caption to condition the language model on visual inputs. Uh see to condition the language model on visual input. What does this mean? This is language model this whole thing but we are conditioning it to visual inputs using cross attention. So for that we insert a new cross attention layers between uh existing pre-trained and frozen language model layers. The keys and values in these layers are obtained from vision while queries are obtained from language. They are followed by dense feed forward layers. So that is the reason why this is called as gated um x attention dense why dense because there is dense feed forward layer this one. Y cross attention, X attention because it's there is cross attention, Y gated because there is stage gating here and here. Okay, so that's why these three terms are there. Please pardo