Small Models, Smarter Learning: ICL

Key Takeaways

hello Community you might say hey why you want to make small language small smarter well just two days ago I posted here on my community post here that I read here that massachusett Institute of theology published open eyes announcing that its technology will be deployed directly on the battlefield and MIT tells us here defense contract completes here its military period so okay December 4th and and really the next day it was here the second Surprise by openi that they now offered that we can pay openi for their reinforcement fine-tuning of their 01 models in the first quarter of 2025 and I thought we understand exactly what is happening here in this reinforcement learning based F tuning because it was kind of invented here by Princeton and UC Berkeley now it is open source so why should I pay this particular company that gets now he old the money here from a different source and I decided here that I will not give them my money anymore so therefore here you understand why I suddenly have the topic how to make small language model smarter so you see if I focus now more on open-source AI models and therefore on smaller AI models that I can run here on cloud infrastructure like Lambda or other Cloud providers there is now a real simple question I have to ask answer how to make a small language model smarter so I use all my intellectual power and let's find a solution so you know here we have here survey on in context learning here by ping University Hong Kong poly technque and con Mal here and it is due to its zero training cost that I think this in context learning also know as a few shot prompting this has really gaed some attention here and emergency as a promising approach to imp proof here the reasoning capabilities especially of small language M and I think this is a beautiful publication if you're interested to this because the key idea is simply if you think about a human it is kind of an analogy based learning now we say hey I give you some example and now come up with a solution that is kind of similar to the examples I have given you in my prompt now it expects here this llms to discern hidden patterns from carefully Crea demonstration examples 1 2 5 10 and subsequently generate here the appropriate improved reasoning step for unseen test problems so of course you know that here and this is another publication my University of Texas John Hopkins University and Princeton University chain of s he see a pivotal advancement here in the reasoning domain and by incorporating the simple prompt let's s step by step here alongside and please do not forget those they stepbystep reasoning examples you have to provide this approach enabl here our AI models or a small language model in particular to kind of emulate your humanlike reasoning processes now in this publication here by NYU large language models are Ser or time series forecasters we understood the language model appear to have accurately here the components to do this time series analysis such as linear trans combined with seasonal fluctuation the systems kind of somehow understand here the complexity of a Time series forecasting but of course we also understand for this publication here University of Edinburg and and other company that we have you to be really careful here because I based reasoning performance is highly contingent upon to provide demonstration examples so let's talk about this for a second this llm was found out here particularly in this uh very interesting o preprint exhibit here High sensitivity to the task specific characteristic and the multiple facets of ICL example that we provide in our prompt to the small language model and this characteristic here include here the demonstration quantity 135 or maybe 100 maybe 500 then the ordering of those and the label distribution you're not going to believe it the effect of a label Distribution on the ICL performance of small language models so when our ICL encounters here a reasoning task that shares similar logical structures you see this is it no because if it differs here in the presentation format that we have here the the formal structure of the presentation in our promptu our mod language model reconstructing now additional correspondingly structured demonstration example is sometimes really necessary because we want then to have an unseen task but therefore we also have the same presentation format of our few shot examples in our in context learning to receive you the maximum benefit from in context learning now just yesterday here from open id1 system card if you studied this I've given you also here the hint in in my community tab of my Chann here this chain of sord deception monitoring and here op tells us you know if the yeah and I say if thei is lying to you if it's hallucinating it's inventing incorrect facts incorrect rules suddenly such Behavior could plausibly emerge from the 01 models here the reward hacking during the reinforcement learning by human feedback during this alignment step and this this is interesting because now Opa tells us there's a possibility and I mean if open tells you this year in the official documentation you understand that it is not only a theoretical possibility that optimizing our One models to prioritize your user satisfaction couldn't result those models here providing overly agreeable or inaccurate responses so reading this I understand that whatever happens here in reinforcement law morning in the alignment phase of our llm that the llm behaves in a particular way that we force the llm to behave and if we do this in a way that is not in coherence with the other data sets from the pre-training and the fine-tuning then we're in deep trouble because then we see a massive hallucination or it could be theoretically a possibility that we have inacurate responses so what this means is that the qual quality of the pre-trained LM in particular here the form the data structure the complexity and the presentation of the pre-trained data set is from utmost importance especially if we apply here reinforcement learning algorithm where the data set is not coherent to the pre-trained llm so any incoherence we have now the training data set between those two steps may lead to an increased hallucination because the LM simply doesn't know what to do with those inconsistent data sets now the same is true and we already found this out if you have a pre-tuned llm and you just fine-tune it in the old times we sorted we could fine tune a model to a new domain knowledge today we know that this is not possible if it's not in the complexity in in the inherent domain structure of the pre-train data set the fine tuning it will modify nuances here little details but you cannot interact here complete foreign domain knowledge only by fine-tuning the performance of the model will go down down and down this also means that if you have your structural or complexity or theoretical incoherence here in those training data set from the pre-train to the fine-tuning data set you have here an increased theoretical probability of hallucination happening in your fine tune system or your reinforcement learning align system and now you understand why in my last video where I talked about Y apoj 2 the latest Vision language model by Google if you look they give you here only the pre-trained model and some of you ask me hey why suddenly we are back to the pre-trained model why not fine-tuned why not DPO aligned What's Happening Here well they did here and if you read the documentation carefully they have here a high quality pre-training data set and they give you the the details and you see this in my video on the pre-training data set so we know exactly what the pre-training process included and now we can start for our particular visual data set toine tuning process we have to adjust this otherwise the performance of our vlm will go down and as you know also Google provided us with some fine tuned uh version on particular visual data set like for x-ray or for medical application but those are highly specialized fine tune systems and of course as I showed you in the video I or here hugging face also gives you here exactly the fine-tuning notebook how to do this and you see here pojama processor and from Google here you have pjama 2 the three billion pre-trained model only for 448 time 448 pixel you see this is now the Insight that it seems that the industry is understanding that we have to have a good pre-trained model and then you build with your particular data set on this high quality pre-trained model coming back to in context learning this is also really really important for our in context learning because in this video I showed you here that ICL can outperform your fine-tuning or even re systems and especially if you use some unsupervised ICL plus like in this video you get a real good performance out of this system we have here this publication from this is from I think Google Deep Mind yeah completely where they go here for many shot in context learning so if you increase the amount of in context learning few short examples you provide then the quality goes up and Google Wes here we find that both reinforced and unsupervised in context learning can be effective in the many shot regime particular on complex reasoning task so another indicator that this ICL is really I mean it costs nothing in the pre for the training data for the Post training we just have to find here to write adequate few short examples and they say you know you don't have to do the supervised ICL if you do the unsupervised ICL this means you just provide examples of the problem in your prompt to your machine to your small language machine without giving you a solution in the prompt without giving the solution in a few short examples this can improve your model performance substantially and if you ask yourself howest this possible even if there are no labels no solution available for commentar like mathematics here the mere presence of the relevant information may help here the small language mod to CU here the skills and question and answers will only be strictly necessary here so to label data if you have a task that is truly novel and was not at all part of the pre-training data set so if you have a good pre-trained data and you use your unsupervised ICL you just provided problems but problems in the same complexity or maybe in an increased complexity because you remember the ordering is important here you don't have to have all the solution you don't have to have the labels to it but the mere prescy of relevant additional information may help the system to come up with Advanced reasoning processes which is amazing in its own rights let's look at this publication here it's called a mystery of Inc context learning here a comprehensive survey on interpretation and Analysis and they go here for a more mechanistic approach here highly recommend this also it's already in version three so it's more than a in the original paper more than a year old but it is highly interesting because they analyze here the effect of the pre-training data properties on the performance of the ICL and they have a real deep dive into the pre training data properties and they demonstrate here that the llm so in our case the small language mod even more they cannot perform here a new task through an ICL if the diversity of the pre-training task Falls below a certain threshold so as I told you the pre-training data set the quality the diversity the complexity and even the ordering are of utmost importance here to have a good pre-trained model that we then can use in context learning for our particular task and they found that there are three critical distribution properties of the pre-training data the drive here the in context learning here and particular the investigated to the role of the induction hat in our Transformer architecture and they show this are crucial components for implementing here the in context learning and you know the induction had to copy and propagate the token patterns enabling you the sequence prediction by leveraging a prior context and they also found that models with deeper architecture or higher the induction head which is quite complicated to build here in a small language model but nevertheless we could maybe try to implement this on a single layer basis a little bit later in the architecture demonstrate your Superior in context learning performance reinforcing here the connection between depth and learning efficiencies and if you're not really sure hey what is in a transform architecture what is this multi-head s attention attention mechanism and how does it relate here to this special hats short summary the induction had emergency as kind of a natural consequence as they write in the paper of the training process of the pre-training process when the Transformers is exposed to data containing here repetitive patterns even hidden patterns or different associations on the semantic level so induction hat just one of the many roles here an attention hat can take on within a multi-head self attention mechanism of our Transformer of our classical transform former architecture if you want to learn more about this in context learning in particular the role of this induction had kind of I would say have a look here in March 2022 and Tropic published here I think one of the first main studies here what A induction at how could we detect that there is a specific specialization going on within a transform architecture and yeah an Tropic showed us the induction had emerg as attention ads that specialized in recognizing patterns coping with patterns copying pattern extending your patterns and maybe even understanding here specific sequence of patterns now let's come here to another publication that's happened just yesterday this is here by Google Deep Mind and UCL and they now and it's what a coincidence no that they now investigate here the broader spectrum of in context learning here they go in general for large language M but I would like to focus on the aspect of of small language M with a little bit of modification but they rather easy and they think about here how to improve in context learning you remember zero additional costs we just need a good pre-trained model and if you have a highly specialized domain specific then you fine tune it a little bit but otherwise ICL should be it and it costs nothing so this is now a really interesting app application here for our next step so they say adapting to task from instructional role play we even extrapolating here the time series we already talked about the time series instruction role play this is now interesting because this shines now some new light into the in context learning mechanism so highlight the importance of generalization such just can be studied along several mention not only the ability to learn something novel but also the flexibility in learning from different presentations and in applying what is learned so let's have a look this goes here if you know the term meta learning it's about one two years old here to our goal conditioned EI agents Google here is of course focusing also on an agent application specific what whatever so in context learning can now be categorized as an element of something bigger in context learning as the ability to use the context of early observation and a sequence task to support now prediction by our small language model or decisions taken by our AI agents later in that sequence in the time series in a ways that it is a meta learned across here distribution of sequences so you see we bring here in context learning Under the Umbrella of meta learning intelligence and if we Define now this sequence task to be any task at all involving making a sequence of predictions or action if we have an agent based on a sequence of observations with our robotic friends of course here sensor input then we're talking here about a sequence task now the relation between the observation and the actions within and across this different time step evolve according here to a latent process but we know this now the lat processes for each sequence or sample from a real broad distribution that may have some Universal consistent features as well as some that might vary with time or with conditions and this you notice if you have subscribe of my videos you know this definition fits perfectly what a coincidence with the partially observable Mark of decision process that we already examined in multiple of my videos here especially in the reinforcement learning rning regime and in language modeling so you see we have now if you want a new pair of goggles a new pair of glasses we put on and we understand that this in context learning is here really connected here to our mock of decision process where we know and we have all the mathematical formulas and the mathematical tools to operate with those and now we applied so the full spectrum of this meta learned in context learning ranges here from a simple memory like we have in with our agents so basic use of context to resolve linguistic dependencies to supervise F short learning and onto much more complex in context adaptation of our small language model and this is the beauty this opens up here a gate way here to some more complex in context learning and maybe we even understand what is going on now you remember we talked very briefly about instruction following instruction learning so this extends of the Paradigm here of ICL Beyond specific examples like input output pairs in the future learning regime to now complex scenarios where the model adapts to explicit directive or task description so this means that now instruction following if we see it with this new polarization doesn't depend on specific input output examples for the task but the model this small language model adapts now here based on the semantic of the instruction itself so unlike here the traditional F shot task that we provide in our prompt where adaptation is constrained here to real specific examples in specific ordering in specific form with labeled data or not label data instruction following allows now for a more Dynamic adaptation to entirely new task and yes you can feel the excitement because instruction following showcases how eyes in context learning here extend beyond the pure data pattern regime that we have currently to more abstract task description and I think this is the way forward if you want to use your open-source small language models and have a better reasoning performance on those mods and beautiful we have finished the introduction end of part one and tomorrow we will talk about the real implementation here of these new ideas how to improve in context learning for better reasoning and causal reasoning performance of our small language model that hopefully are open- sourced and we can use it locally