Fine-tune ChatGPT w/ in-context learning ICL - Chain of Thought, AMA, reasoning & acting: ReAct

Key Takeaways

hello Community I hope you are impressed by the title but you know the truth is it is very easy today it is one of the simplest and one of the most complicated things I would like to show you why this video I want to show you if you do not have the money to run the fine tuning of GPT systems or you cannot afford find your own chat GPT or you cannot afford fine tune pom540b or whatever Auto regressive large language model you have there is a way that without spending tons of money actually you don't have to pay anything how to get the best results out of your pre-trained llm and it the answer is simple on the one side you have to give an intelligent input but it is something completely different what you might expect and with this input you get significant better results back without spending any US dollar at all so for this we have to do a little bit of theory now you know the fine-tuning the word fine tuning uh I have about 50 videos about coding fine tuning on bird models biobird date of birth or whatever you want and then on the other side of the Transformer on the decoder stack so to speak we have those GPT monsters gpt3 bio GPT jet GPT and now they also use the term fine tuning now what I would like you to understand and you are done one of not a lot of experts that fine tuning is not always what it is with cheap adhes system we can do prompting and for example I will show you with bio GPT we will do a prefix tuning officially or to be unprecise everybody even Microsoft calls it fine tuning but I wanted you understand if you have to code it it is not fine-turning it is something completely different and I would like to show you how to do it and give you the reason of the explanation why it is so gpd3 or jet2pt or whatever you want to call it there's the option of prompting now prompting means pre-pending Specific Instructions and a few examples to the task input and then you are generated the output from those language model which is significant better so the system learns with an intelligent input so gbd3 in particular uses manually design prompts to adapt its generation from different tasks and this framework has a specific term so when you hear in context learning or ICL you know it is about prompting input prompting prompt engineering the input to a GPT system now there's a limitation since Transformer and we have here now the decoder stack you know can only condition on a bounded length context where gpt3 it is defined with 2048 tokens in length so the in context learning so this is what I will retry with prompting is unable to fully exploit larger training sets training sets that are longer than the context windows so you have a quite hardcore limit on 2048 tokens for GPT 3 for this condition therefore and I showed you in one of my last video that bio GPT was now released by Microsoft on the hugging face platform the model they diverted a little bit and they used a recent development that is called prefix tuning now this is a lightweight alternative as they themself call it to fine-tuning for a natural language generation and we are here at autoregressive system so you have given the beginning of a sentence and you want the next word and the next word to be inserted by the machine and this this new methodology was inspired by prompting so you see you have the classical fine-tuning task on the bird system on the decoder and on the encoder you get prompting which was really nice as a new alternative and the latest state of the art if you want here uh biogpd just some weeks ago or days ago it was published on hugging face you have this pre-fixed tuning and even Microsoft calls it a fine tuning approach but you will see it is not but what we can learn is we understand the system and we can get results even without applying fine tuning if we have a clever input so here we go whatever we tune whatever we do we have to have data so all these models whatever there is they are pre-trained on a huge data set millions billions trillions quadrillions of documents beautiful if we do fine tuning we do it for specific Downstream task let's say today our Downstream task is question and answer beautiful now the first thing that you know gpt3 is for years now available so you have to open your API fine tune create and then everything and if you read really clearly here and please copyright information this conditional generation on openai you have your paraphrasing summarization entity extraction product description chat Bots whatever you can imagine what it is and they give you guidelines if you pay uh openai to fine tune the gbd3 model for example say you have to use certain separators you have to have an end separator they are aim at least for 500 examples and I show you exactly how the examples have to look like and they tell you hey to ensure that a prompt and the completion does not exceed the 2048 tokens including all separators so everything as I just showed you here we are bound by 2048 tokens here for LGBT suits for example this is a example for gpt3 so please check yourself I have here a platform on my eye dot guide fine tuning prepare your data set so this is the classical approach and here I would recommend going to uh this notebook here I leave here a collab research link fine tune gpd3 with whites and biases with w and B this is a really interesting page have a look at them I'm not sponsored by them not at all but use the resources that are available so and here you see for example what is the prompt and the completion structure we have here the prompt of prompt one let's say it's Livermore California the completion Livermore is a city in Alameda country California in the United States and beautiful next one Albany Woolen Mills also known as Western Australia Western and wool Mills limited was a wool Mill located in Western Australia and so you see you have a prompt and you want the system to complete this prompt with this learned information so rather easy to understand beautiful now this was gpd3 now let's look at biogibility you see here published an archive January 13 2023 Microsoft Microsoft Microsoft I had a particular video already on this but what I want to show you we need data for biogibility and I want to show you the data Microsoft used and this is a public domain data so it's called public medical q a and this is a biomedical question answering data set and as I told you we are today only in the question and answering part to make it easy understandable so in this public data set that is available you can download it you can play it you can use it you have a sample structure so you have a question a biomedical question what is the drug what is the biochemical structure what is the molecular structure how is the drug drug interaction whatever then you have an answer and from the scientific papers the abstract of a scientific paper you have additional information that's called a reference or just a context and then you want as it's an answering data set they go for a binary or tertiary answer yes no maybe this is what they call the label so you have question answer context and label in this structure where the data set is built up in exactly this particular pattern so and then you have to know two information to further information you have the source sequence and the target sequence and I will show you both so PubMed is of course the first question and answer data set where they tried reasoning over the biomedical research text especially the quantitative context yes yes yes but why we well Microsoft uses this it is the largest size of expert of human expert annotated yes no maybe question in the biomedical domain so we have here the human input human feedback the human evaluation of this data set so if you want to have a look of course there's a research paper please have a look here you're not going to believe that Google was involved in this but of course and here you have now one data set just to show you the structure and we need to understand this to generate better input so at first you have a question beautiful and then you have the context from somewhere an abstract scientific abstract they tell you blah blah blah yes beautiful and then they have what they call here a long answer so some were in a scientific paper there's the sentence I was studying indicates that the pre-operative state and therapy seems to reduce some development of some whatever so this is the conclusion because you ask is there something and they said yes and now from this long answer a human expert came down to a binary answer yes no oh that's right yes no maybe when this is exactly what you want but just to tell you have a look here a study indicates that the therapy seems Seems to reduce so I would have gone here before maybe but the expert decided to go with yes here in biomedical so beautiful so you see this is the data structure that we will now do the fine tuning on and as when you read the research paper Microsoft writes the fine tuning for Bio GPT for a downstream task remember we are always focused on the downstream task we have a question answering or a question and answering whatever you want to call it fine-tuning here is if you really are correct we have to tune with the code it is the wrong answer because it's not fine tuning let me show you why so again this is the text here from their publication and you see they have a source sequence and a Target sequence and now say they are methodology from January 2023 here to fine tune bio GPT Microsoft itself is a we prepare the description word question context and answer before the question context and answer and concatenate the string together as the source sequence so you see here question context beautiful so they have now a sequence of text the question itself that you want to know then in the fine tuning they provide you now the context from the scientific papers a summary or the whatever particular Parts particular chapters of the publication this is now that goes into the context and the answer is taken from a publication now for example the long answer the answer text this sentence here and then you have a Target sequence and you want this target sequence to be structured in a particular logic so you say the answer is answer text to the question question text given the specific content content text here is and now comes the binary classification problem if you want yes so this is more or less the input data For What Microsoft calls fine-tuning bio GPT but we now know that it is not a fine tuning but it is a prefix tuning because if you read their paper you come to the paragraph where they say what is absolutely correct during the inference run so after the pre-training after the fine tuning if they have some in the interference run we provide the source text and the prompt as the prefix for the language model to condition on and let the language model generate the target output so you see this prefix here in our sequence where you have question context and answer all these additional information you feed in your input to your query this is what they call prefix tuning and prefix tuning optimizes a task specifics prefix that applies to all instances of that particular task and this is why it is used as a advancement of fine tuning because we have Downstream tasks and here with an auto regressive system we have the beauty that this applies to all instances of of the task it is Task specific so prefix tuning can be applied to nlg tasks so if you now code this this means that the language model parameters remain fixed and just the prefix parameters of the system become now the only trainable parameters to give you a rough idea of Dimension if you say the whole model parameter is 100 I suppose as an idea the prefix parameter is about 0.1 percent so you see the amount of training you have to do is significantly smaller but you get similar results now of course this prefix tuning was not invented here by Microsoft but you have the original scientific publication here by Stanford University prefix tuning optimizing continuous prompts The Prompt input for generation for text generation have a look here at the archive preprint I can really recommend this paper just to give you an idea between the technical term fine-tuning that we apply here and the methodology of pre-fixed tuning that bio GPT by Microsoft was fine-tuned on it is the coding of a prefix tuning and you see here normal you have the Transformer with its layer and you have the fine tuning and everything is is in motion referencing will be altered all the weights will be included and so on for the task of a translation for the task of a summarization put a task of table to text or for the task of question and answering now with prefix tuning as I told you oh there's weights of of the model stay well if you want Frozen you have just this small Vector here in front this prefix and this is where the encoding the embedding is gonna happen so from this publication optimizes a small continuous task specific Vector here and this is called what they call the prefix so therefore prefix tuning it is a task specific Vector tuning based on your content of course now if you dive a little bit deeper in this paper you get an idea you get a little bit of mathematics that it tells you how does it work with the loss function and trainable parameters and the differences uh not for the moment I just want to give you the main messages despite learning 1000 few times fewer parameters than fine tuning the prefix tuning can maintain comparable performance and this is if you think about a theory amazing one thousand times fewer parameters of a model and you get the same performance if you just do some different tuning of the model so this is one of the advances here at the end of 2022 beginning of 2023 that also you should take advantage of because if you have to pay up mail for fine tuning and you do this methodology well you can calculate the difference there are a lot of other options coming up now you see here November 2022 ask me anything Amma is the short acronym this is a strategy for prompting language model and it was a very funny have a look at this paper they just say hey let's why just come up with one prompt let's generate I don't know a set of prompts and put all together and try which is the best prompt so you have so many possibilities to play with these systems so they collect multiple effect if yet imperfect prompts and aggregate them and this can lead to a highly quality prompting strategy so if you have no Financial limitations you can try experiment whatever you like so this was more or less here the green part we learned about fine tuning we learned about prompt engineering and we learned about this new technology that advances on fine tuning was inspired by prompt engineering and is now called prefix tuning of large language Model Auto regressive models and now comes the main part of the talk my goodness you have new options with this you can save a lot of money not paying a cloud provider to find you on this on your specific company data of your domain specific data but you can use something more intelligent and it is called em the chain of sort or react and I would like to show you this that you can apply it and you can save something so First Chain of Thought this is January 10 2023 Google research not coincidence um large language models so Auto regressive models chat GPT whatever comes out of Google in the next days weeks months I have no idea there is this beautiful Jane of sort prompting methodology let me show you how it works if you want to read the theory the mathematics I give it a link to the paper but just to show you what it is you have normally your standard prompting you have your model and you say you type in hey this is my question and you get an answer back by the system and then you have the next question and you get an answer back and now this is time the answer is wrong now they find out if you provide not this information these three lines and an answer in three lines and an answer but if you have here this is the input text now to this question so you give an example a complete example as an input to chat GPT or gpd3 or whatever you have you say hey Roger has this that says how many and then you also give the answer as an example and you give the chain of sort your reasoning your way your path of reasoning and you say okay so we start with this and then we put this and then is this is the result here first step and then we do this and then we alter it and then we get a result and given that the answer the structured answer here and you have a similar question now here as the next question you really want to have solved of course they have to have some conics here you can go over some completely crazy here you have to have some overlapping if you want you will get an output where the system now follows your I will not call it reasoning that is too much your path of augmentation your chain of sort how to come to this result and this is something beautiful so you can teach the system if I have this question I I want from you this kind of answer this is the way how you have to construct this answer for me and now I have my question the model really and this is the beauty of Auto regressive systems comes out and learns this here now there are huge a lot of question how is this possible why is this possible what's the mathematics how is the configuration of the system what are the parameters forget all about it read the paper just want to show you what you can achieve what you can do without paying extra if you see but what's the difference let's look here just at the yellow and the kind of orange here and this is the world of mathematics and you use yeah we know we have to use huge system so Palm 540 billion parameter system you have here if you use Palm the same system in yellow here and you have the standard prompting so you just have a what is I don't know three plus five and the solve rate will be just 18 percent although this model has been trained I don't know for weeks months centuries it is it is really on mathematics just this is the sole rate is is under proportional let's call it under proportional now the same system you don't have to pay for any fine tuning on mathematics you don't have to pay for any additional modules you don't have to pay for any solver packages that people sell on the internet if you just reformulate your input you can get from 80 sulf rate close to 60 percent just by asking or providing this kind of information as input to the system so you see this is something you really should make use of if you interact with that GPT weather systems and they are able here to have these jumps in performance because you give them a chance to show you this is my question this is the answer I expect and as a a rule of as a feeling I would say how many question answers you have to provide to really achieve a kind of an Optimum and in my case now for my text and for my ideas and for my answers it is about five pairs of Q and A that I write q1 A1 Q2 A2 so till I have five and then I have my main question some colleagues tell me I need eight okay so but between let's say between five and eight if you give five to eight examples those huge system with 540 billion parameters they are able to answer you in the way that you want it so do not stay with one example but just as a rough idea five to eight and then your question and you will be amazed by the quality of this answer so prompting a pump 540b just eight chain of Swords excellent Chief state of the art you have to a different mathematical Benchmark and recent so process even a fine tune GPD with a verify and whatever so you can do something if you have a clever input you have to know about this and you're not going to believe it now we have the next level of complexity what we do now we take this chain of Sword prompting and we combine it with acting and acting as simply as I showed you in in u.com chat and in perplexity I you integrate now online sources like Google search or Wikipedia or oh yeah bing bing will be the new you know what I mean so you have now this chain of Sword prompting with the ability that the system has now here this link to external information and what is this called react maybe you will hear people talking about react and this is not about something what you might think of but this is simply a Synergy of reasoning and acting reasoning and acting in large language models uh Google research yeah well what a coincidence and Princeton and they combine this and this is the research paper please have a look indeed about the react the reasoning and acting in large language model just gives you some ideas so we combine our hopeful abilities for reasoning as Chain of Thought prompting and acting and yes yes yes so knowledge base and I just told you everything the nice thing here is if we now look also on question and answer this is the easiest topic you also can verify just seeing the results you see here that you have significant better results the research paper gives you applying react outperforms imitation and reinforcement learning methods and reinforcement learning is a big part of chat GPT by an absolute success rate of 34 and 10 so these are really Champs that you achieve in the accuracy in the success rate of the system when you prompt a system in this react way and as I showed you it is rather easy to implement it you do not need any code any specific you just have to know how to do it so yes yes yes text text text uh let's have a look at example so here we go you have a question the question here is aside from Apple remote what other devices can control my Apple remote beautiful now if you have a GPS system you get a standard reply an iPod which is okay wrong beautiful if you then just apply this chain of sort and you have here this input and to this question and you get an answer that is has a multitude of answers element but is also wrong and then if you take from react only the Act only so you do a search and this is more or less what you.com would be or perplexity I would be when I showed you that they have access to Google search or to Microsoft Bing the search engine you see here you have an act one the search you have an observation of the agent that is now getting some additional information from its environment you have an Act 2 another search if this is not does not deliver the results you're looking for you have a second observation and then somewhere you stop and they give you an answer and the answer is wrong and now this the beauty is of course if you combine reasoning and acting so thinking and action together they give you here this example where I say sword one I need to search Apple remote and find a program it was originally designed for to interactive wave so you have an action you start action one you search or apple remote on a search engine you get some some paragraph back some some text back and then you have your thought number two apple remote yeah yeah so you search for a more detailed question you get the observation back hey I could not find this so say no problem if you did not find this we need to search for a new I don't know a term topic whatever and now this the act of search happens the observation comes back is discontinued and then you know exactly your sword number four and this finally is the sort that brings you to the finish and gives you the correct answer so you see you have this kind of reasoning action that you present to the system so that the system can learn from this think about it as a it's a a cookbook it's a recipe for an llm for Jetta jet GPT or whatever Auto regressive systems you work with this is the recipe you tell this system in the input prompt Hey listen this is my recipe do it I think this is the easiest way yeah there's decision making I would not go to decision making if you want to have an idea how the the the theory behind it is is simple they say that within the agent they augment the agent's action space where they give it now and another dimension or another space in itself which is the space of language so an action high is now part of the language space which we will refer to as a sort or a reasoning Trace but this does not affect the external environment so we have no observation feedback from this but we have extended our now think about a vector space so we have more or less more possibilities and they say as the language space is unlimited yes yes yes and we have a poem 540b is prompted with few shot in context examples to generate both domain specific action and free form language sorts to ask to for task solving and it's nice it's it's a little bit challenging yeah they have some nice results but I would say have a look at the data yourself just to give you an idea this is what I wanted to show you with the last information from this research paper here um if you have the learning now as a prompt or as a learning at the classical fine tune they show you here the scaling results for prompting and fine-tuning on a specific fine tuning always specific task in the task is question and answering if you use notice react scheme and they have your different models and you see here you have here in blue the standard model in Orange you have the chain of sword in green yeah activate it and then you have here in red that react way and as you can see here the model size this here for example is a 540 billion Palm model and you see independent of the methodology that you use here you are about I don't know 25 whatever uh that that percentage indicates on success rate or normalized success rate or whatever but you can now and look here you can achieve here with the real fine tuning of course I mean if you really have a huge data set that you can find in the old system on you achieve even with the 62 billion Palm model you achieve here let's have a look at the react the best result of those four methodologies but even higher than with the 540 billion model where you just have the learning methodology as in the prompting of course if you have real high quality data thousands and thousands and thousands of elements of new data where you can run a fine tune of course it is better than when you do just eight examples in your input prompt for a pre-trained llm but just to show you so we fine-tuning if you have the chance to really find you in a system if you have the data the human generator data wherever you collect your data fine tune is the way to go even with react you see even you can come down from a 540 billion system just to adjust 62 billion size system in Palm and you get even better results with fine tuning this was it for here I don't want to go in further details yeah there are different Benchmark have a look at this it is really interesting um yeah and now there's something else and this will be the last part of this talk now today I want to show you now that you have seen that with the structure of your input prompt the structure of the question that you do not just have a question but you provide information before you ask the system so that the system can learn on this little place when you ask something how how is this possible now I showed you that it is possible but what is the deeper explanation of it and there is a beautiful paper Google research MIT Stanford and they try to come up with an explanation and it's a mathematically a little bit challenging but really interesting paper please have a look at this and it is the question why is in context learning in Auto regressive systems like large language model possible what happens how can you simply by providing some input some demonstration in the in the question field and the system learns from this without being fine-tuned in the heavy old-fashioned way and the answer they come up with is that there's a system within the system that the the hidden layers have if you want a self-similar I will not call it a reasoning structure a self-similar connectome developed so that very simple tasks like regression can be performed although not being exactly fine-tuned for this task but just given five to six seven eight examples and this is for a theoretical point of view a fascinating question but this would now I think be a little bit too much for today have a look of the paper it's about 10 12 pages of mathematics if you like mathematics go for it I say thank you I hope you enjoyed it a little bit and I'll see you in my next video