Inside the VLM: NEW "Task Vectors" emerge (UC Berkeley)

Key Takeaways

hello Community Welcome to our Halloween edition do you remember Albert Einstein when he talked about spooky action in the distance well today we're going to look at some spooky action in artificial intelligence so let's just jump right into it and we are talking about Vision language model and we will have a look inside the vision language model how their transform a layer work on a particular topic and we are going to talk about a task vector now if you're not familiar with this this is the study I found the most intelligent one in context learning creates task vectors Google Deep Mind almost one year old and if we Zoom a little bit in here and they tell us here this in context learning in the llms has emerged as a powerful new learning Paradigm a year ago and they say ICL can be seen as a compressing here as our training set of the I into single task Vector t s and then using this particular task Vector to modulate the Transformer to produce the output and this at the time was a brand new idea and if you think about it let me put it in a simpler wording so we do have now in our highly 1,000 dimensional 2,000 dimensional Vector space we have now by our architecture of our transform architecture designed the task VOR and this task VOR is now a compress representation of the information contained in the few short demonstrations s of our incon learning example remember in a prompt when we have an i we have one two three examples and then we have the query now it is important to understand we are just talking at a moment here only about this few short demonstrations called s and then this is computed by the first layers I don't know four or five of the transform architecture if effectively encoding the mapping of the rule exemplified by this fure demonstration s and you might say okay this sounds logic however note that the query in this particular publication is not directly involved in the computation of t s so a modulation is now implemented by T patching t s at a specific layer let's say 12 of the Transformer during a forward pass with together with the query X and now the remaining layer of our transform architecture then operate on both they operate on the query X and on the newly patched Theta s to generate the output and people at the time were really amazed to say how is this possible that we can have if you want a complexity reduction of all 2 3 4 five in context learning example into a single Vector thet s now here study end of February 2025 they found function vectors in large language mods and they approached it a little bit different but also we're looking here in context learning and they said we test semanic Vector composition here in function vectors and find that some of the extent can be some to create vectors that trigger new complex task so solving the task internally into a in a language mod we had the creation of these new vectors and you might say is this limited only to language mall now we come here and we have a happy little ghost here for Halloween and he is here examining here the visual prompting here this is UC berky Google research October 7 2024 and they find this now also here in the visual llms in a vision language model and they find Visual task vectors deep inside the vlm and this confirms that task vectors exist here in the network activation space and they can guide the model to perform the desired task also in VM and with this let's call it a breakthrough to understand what is happening here in the task definition inside a vlm or an llm now we have a new research and they tell us hey inside this realm task directors you know what they are cross model now this is a little bit spooky because why and here we have a brand new publication just from yesterday October 29th 2024 from UC Berkeley a beautiful presentation you have to read this after you finish this video what are they going to tell you the DAT discovered they said okay we looked at division language mall and what we found if we have multiple input modes let's say we have a visual in context learning example this is our prompt we have for our vlm the image of a flag plus in the pair we have then the capital of Paris of France this is Paris or they go here with a textual instruction so instruction not tuning but providing the prompt in an instruction form and simply the command is say map the country to the capital and then they give also here textual in context learning examples two for examples France keep all your pairs France Paris Greece Athens and you understand it so different specification here in the input on our in context learning few short examples mixed vision and text or an instruction or purely textual and the funny thing is deep inside the embedding space here of our vision language mod and this is here in a multi-dimensional space here some hyperplane or a manifold and they found that in this space all those things that go together cluster together this is fascinating because why should an image task representation be close to a textual task representation even if the task is about the same topic but hey do image not encode differently in high dimensional Vector spaces well it turns out not at all and this is the beauty of this study so in simpler words a major Insight now for UC Berkeley yesterday was the division language model map conceptually similar task like I showed you mapping a country to its capital to similar task Vector representation so this task re becomes now here the prime object of our interest and the embedding the representation in a multi-dimensional vector space and this is regardless of whether the task is specified through the text examples that I showed you here or through instruction or visual ICL examples and you might say how is this possible how is it possible that in the same region of a space in this High dimensional embeddings all those three text instruction images encode in the same narrow Subspace region and why so let's have a look at this what they found is that this cross modality Vision text instruction capability allows here the task Vector defined in one modality to be transferred since it's in the same Subspace no to another modality and they look the text to image and vice ver and this they found out enhances here the task adaptability in this models so let me give you an example again I stay with the simplest example possible country to Capital so in text as I showed you France T value Paris France Paris or visually as I showed you the French flag and Paris then as the label with each format now leading here to a similar representation of the task as a task vector and this means that this shared representation enables now vlms to handle cross model tasks and this is significant if we apply this because what we have now is an inter modal transfer now imagine you have your classical llm beautiful and then you have your fine-tuned vision language model now for the same text ICL inputs the base llm and the fune VM contain highly similar task vectors and if you calculate here the coine similarity of the task Vector in the LM and the task Vector in the VM you find the real close almost one 0.95 is their coine similarity and this is the reason why the llm task factors they can be patched into the image query of the VM can you imagine what this means you have the capability of an llm to perform a specific task and you have a VM which has the image conditions and then you can extract the task Vector of this beautiful llm and just patch it at a specific layer in the Transformer architecture to a VM and the VM is suddenly able to solve the task I find it absolutely beautiful yeah please note here in the fine tuning here this was fully fine tuned and you might say why do you notice this yeah it was not Lura fine tuned and if you want to understand here why I mentioned this in my last video I talked about the differences here in the spectral properties of the weight tensors if you have a full fine-tuning of your pre-trained model or if you just have a Laura adapter on top of the pre-trained weight structures this here now what a coincidence that this was in my last video this here is important because they do here the full fine tuning because the question would be in doing the lower fine tuning the path the parameter efficient fine tuning you understand that we have now with Lura the vector composition in a lower dimensional mathematical space so how can we make sure if we have this now where we have an inter model transfer that this Laura element Subspace is available in the VM or can this Subspace at all as an adapter dock here to a vlm task so a lot of really interesting question for eii research if you would have Laura but it is no problem at all if we do here a full fine tuning cycle and not go with Laura all the explanation in this video and we start with a textual ICL the textual instruction visual IC example and you see if you look where they are situated they are all close by let's have a look at the task Vector you see we have here the examples and here we have the query and now we can transfer this to the query it is patched to the query and the prediction is Rome so beautiful so short summary we have cross modal task patching so by mapping out the task vectors from one modality text LM to another Vision VM researcher evaluate if the task defined in the text can be used to inform an image query and they found out yes it's beautifully especially combining here the instruction base modality and the example based modality with the task vectors combining those two enhances here the task representation quality significantly and they call this embling so let's have a look text ICL and we go with country capital indic and let's have a look at the animation you see blue is the input task is here in Orange and the answer is in green you see where the task start at layer 15 and the ons is generated at layer 25 if I go now to food and color you see if we animate this this is the start this is the task and then the answer is generated there's a different distribution across the layer of our self attention mechanisms this is interesting given the complexity if we go for flavor there's another different look at this wow you see between task and answer we have a different distribution here under the probability let's go now to image ICL and you will see it is different let's start with country capital okay wow a very late one and a very late answer to generation but look here at the task this is here takes some 15 to 25 the layer number to come up here with the task Vector interesting if we go now here food and color let's have a look at this so very close yeah oh yeah and then the answer is real stable so this is an easier task as you can clearly see and if we go with food and flavor let's have a look at this yeah it takes a little bit longer oh yeah it's stretch out and the answer it's not so sure anymore you see that complexity is different so beautiful now we understand why we have more or less three faces and I explained this to you for the text in context Ling and the image in context learning but you know there's another spooky thing happening here because if you look real close and you try to understand what is happening and let's say the layer zero here so very beginning you understand we have here a high probability that this is just here the input then slowly slowly slowly here at layer 18 this is layer 18 they examine know exactly what is the semantic content of the tokens in the embedding here of layer in so here they looked exactly what is here what has the highest probability now you understand that here the blue went down so it's not anymore in the input region we are now fully here in the task Vector region and the answering green is also almost close to zero so here we have the pure image or the pure information of the task vector and now we look at this we decoded and we see that this task Rec that layer 18 now has the following tokens cities headquarter City and town and this is interesting do you know why think about it if we have we start here with France and Paris and now they come here with headquarters what is a headquarter a headquarter is the location where the the main part of the company is focused no it is the focus Point here of the complete company in the headquarter they have Personnel Finance whatever HR everything is focused there so you do not have the the production maybe you do not have the logistic you do not have whatever but it is interesting given here the task France Paris the internal spooky thing now is the llm or the VM interprets this here in the token headquarters so if you want the headquarter of Paris of France is Paris which makes kind of sense I'm not really convinced but I you understand what what this is here hinting at so in image in context L it's a little bit different we have a different performance and again this is an indicator that our image ICL is not as good our image model are not as good as our text models so what we have here now we go layer 18 here you see okay the input is gone but we are here also somewhere the answer is here halfway up but the task is yeah task is dominating but you know the mixture is here a little bit more mixed and you see this here know look the blue headquarters here is only about what is the dominant part in layer 18 in the det toiz but it is only let's say one quarter and then you have City Country then you have B strange air okay no idea but you see in the image it is less clearly focused here on this I don't know how to call this this Essence here of the instruction that we give it here with our in context learning examples and this is the internal interpretation by a vision language model headquarters I find it interesting and then of course layer 28 so here we have the green dominating we have the answer so Capital London Rome Lil whatever so this is the correct output but you see this is something fascinating now if you look closer here to these task vectors so again this is here let's say here we look here at layer 18 and we look here at these task vors with multiple uh tasks country capital country currency animal Latin animal young food color food flavor this year our DET toonize task vectors now in plain English this would read headquarter for country capital you notice city city gent or for the currency dollar currency for food and color you have yellow pink green purple orange this looks good for the text I but for the image ICL the same task would mean green yes yellow word yes so you see there is the performance much more sketchy it's not really as pronounced and as clear and as focused as our text in context learning examples and for the food flavor you see it also in text ICL we have flavor taste mild tastes but at image ICL we have yes none anger vegetables so strange so you see if we have now that we have an inter model transfer patching then we can use let's say in very simplified version here the better performance of the text ICL to help out here the image ICL so this is nice if you can transfer here a little bit of the performance of the text to the vision the most significant result in the complete study is that the text ICL vectors when patched for the image task outperformed the image-based ICL baselines suggesting here that the text is much more stable in coding formats for the task vectors now this could be because sometimes our vision language models are based on LMS there are only quite a handful of vision language models where vision and text are really treated here at 50/50 so but it is not really clear if this is the main reason now this finding has implications for designing a more versatile AI systems today as it suggests that unified task representation can enhance the multimodal adaptability a unified task representation this is something I'm not familiar with and I found it interesting that UC Berkeley told us here that this is now a way to construct better Vision language model so instead of having to learn now for our new llm or VM or whatever AI system you have to have to learn each task a new for for each context for example the task is identify color both in a textual and in an image content the model now simply invokes the task Vector that corresponds to color identification allowing it to interpret and response to a wide range of inputs in a resource efficient matter so this means the intelligent multicross modal system does not learn here everything new but somewhere deep inside our whe M the system already learned here for the task color identification that there is a specific task vector and whenever in our prompt we have now this task coming in to identify a color this task Vector deep inside the VM is activated and interpreted and response here to a wide range of inputs whatever the color is red green blue white whatever the system knows how to handle it because of the task Vector now this is interesting imagine this is the core of the intelligence if you want for a task specific query that we as humans give to our llms or WMS so to further understand this let's call it a spooky action here inside the AI I find it FAS fascinating I hope you enjoyed it our little Halloween edition and it would be great to see you in my next video