LLM Self-Taught Reasoning - Explained!

Key Takeaways

greetings fellow Learners now before we get into this wonderful world of self-taught learning I have a thought-provoking question for you when you learn a subject like mathematics how long do you struggle before peing at a solution now when I was in school I would struggle for minutes and hours and I would find my inspiration in the shower or the toilet and this was mostly because I liked math and as an adult I did grow to realize the importance of actually tuning Our mathematical musle in our mind in order to think mathematically and this can come with struggling and thinking about mathematical problems in random places that said if there were time constraints like if I had a test coming up I would just peek at a problem try to solve the solution and then move on to other problems that are pretty similar and solve it on my own now turning this question over to you how long did you struggle on a problem before peing at a solution when learning mathematics I would love to hear your thoughts down in the comments below now this video is going to be divided into three passes where we're going to talk about the whats the whys and the house of self-taught reasoning and also go through some performance and limitations so let's get to it so this here is a large language model and this is going to be pre-trained on the problem of language modeling and what that means is that when it's pre-trained it's going to take in a first part of a sentence let's say I want butter on my and it will try to predict the next word that comes in the sequence that's the problem of language modeling now when this language model is already pre-trained we can now fine-tune it on some tasks and on some of these tasks it performs pretty well so for example let's say that this llm was fine-tuned on the problem of questioning answering where we input a question and we output a response and you can see that typically it performs pretty well on natural language question answering now fine-tuning in llm can also be pretty poor in performance so for example let's say this LM is going to be fine-tuned on arithmetic so if we give it a word problem that involves arithmetic I have five balls but I lost three balls how many do I have left the answer it gives may be incorrect In this case it should not be three balls it should be two so how do we make lm's more performant on arithmetic problems and problems that are not very great at directly fine-tuning well one way to do this is Chain of Thought prompting so a Chain of Thought is a set of intermediate steps of reasoning that link and input to the output and the input could be for example some question and the output put could be an answer now if we take an example let's say that we have a pre-trained language model and we are going to give an example question answer of how we want the answers to be done for an arithmetic case so I have three tennis balls I got three more how many do I have the answer is six tennis balls this is the input example question answer and then we pass in a question over here that's very similar that involves arithmetic but in addition to this we would also inject some reasoning between the initial question and answer so how did we get from this arithmetic question to the answer of six tennis balls well we have some rationale to back it up where it's reasoning I start with three balls and when I get three more balls I add the existing balls so that's three plus three is six and hence six tennis balls so it links the question the input to the answer which is the output and then on doing so the llm will actually have much more context it will be able to generate rationale on its own and you can also see that it generates a correct response in this case and we call this entire question plus rationale plus answer as a Chain of Thought prompt because we prompt it to the llm in order to generate some response that's similar in nature now the pros of using this Chain of Thought prompting is that it actually has pretty reasonable performance in some of these benchmarks it can outperform even when fine-tuning directly on arithmetic for example and also there is really no fine-tuning required in this Chain of Thought prompting case and hence that means less CPU and memory utilization which is all great but we can be doing better with performance and so how can we do better with performance well one way to deal with this is through fine-tuning using rationale or reasoning we can first generate a data set with rationale because we have seen that it helps rational helps and then we can fine-tune the llm on this new data set and this how we actually accomplish this is through a technique known as selftaught reasoning and is these details that we will go through in pass two quiz time have you been paying attention let's quiz you to find out how is Chain of Thought prompting different from fine-tuning directly on the answer a with Chain of Thought prompting the model can generate better rationale B Chain of Thought prompting always performs better than fine-tuning C Chain of Thought prompting makes use of few shot learning while fine-tuning does not or d none of the above are true note that multiple answers may be correct and I'll give you a few seconds to answer this question the correct answer is a but can you tell me why give your reasoning down in the comments below and let's have a discussion and if you think I do deserve it at this point please do consider giving this video a like because it will help me out a lot now that's going to do it for quiz time and pass one of this explanation but keep paying attention because I will be back to quiz you with self-taught reasoning which is a technique it's going to take in a pre-trained language model it's going to take in a fine-tuning data set with just some input and output sets as a normal fine-tuning data set and there will be about if we have like roughly 10 examples with rationale so we can manually create some reasoning that links the input to the output for at least 10 of these examples and that's all really that this star algorithm or technique really needs and on doing so we will be a able to generate a fine-tuning data set entirely so we'll be trans transforming this entire input output fine-tuning data set and generate rationals for all of these examples and we will then also be able to create a fine-tuned llm to generate rationale with the output so that is the star technique what it's trying to accomplish at a very high level now let's walk through exactly how this star algorithm can work so first we are going to have a set of 10 prompts like I said of the question the rationale and the answer and we can use these as like in context prompts and we pass it in along with the question that we want answered in as a part of our data set the LM is going to generate the rationale along with the answer and if this is the correct answer what it's going to do is it's going to add the question the predicted rationale that was made along with the answer and just append it to our existing data set if the answer on the other hand was incorrect we would give it a hint and just say okay the answer was incorrect but we would provided the actual labeled answer this is the correct answer that you need this is the question now we will prompt the llm to generate the r rale so it's kind of reverse engineering the answer and the question and trying to have the language model create a rationale that somehow links the two so we're just giving it like a hint and this process is known as the rationalize step and so when we try to rationalize or reverse engineer the rationale we will generate this new rationale and we will have the answer too and also now add it to our data set we do this process for every single question answer pair that we have and then we will be able to generate an entire slew of question rationale and answer data set so we have that data set and when we have this data set we can then use this to then fine-tune the language model and what that means is that we would then still do the same thing of providing some input prompts of question rational answer along with the question have the model make a prediction of like the rational and answer compare it to the actual answer that is supposed to be the case and then we would actually then back propagate because you know if we compare the actual answer to the answer generated by the model we will get a loss that loss can then be back propagated through the language model and the language model can learn with this new data set and we can repeat this process for multiple iterations quiz time it's that time of video again have you been paying attention let's quiz you to find out without the rationalize step in Star a the llm will not learn from inputs it guessed the correct answer B the llm will not learn from the inputs it guessed the wrong answer C the llm learning will Plateau faster than it would have with the rationalized step or D the llm learning will plate slower than it would have with the rationalized step note that multiple options here may be correct and I'll give you a few seconds to answer this question the correct answers are B and C but can you tell me why give your reasoning down on the comments below and let's have a discussion and that's going to do it for quiz time and pass two of this explanation but keep paying attention because I will be back to quiz you in this past we want to take a look at some performance of using self-taught reasoning and also some limitations of the approach these two graphs show the performance on arithmetic specifically on adding multiple digits so the red line is for on- digigit addition adding two one-digit numbers then the yellow is for two two-digit numbers and so on We compare the star with rationalization to the star without rationalization so with rationalization it's basically going to make use of the rationale generated from the responses that it originally predicted incorrectly so it should ideally in theory give us more signal and in practice that's kind of what we're seeing here too we can see that with just a fewer number of iterations we can even like the harder problems especially like five digigit Edition were able to perform fairly well very quickly whereas for you know without rationalization you can see that it took like 20 to 30 iterations to even get get anything going now other than arithmetic there are some other problems that this rational generation and rationalization help in and this is specifically in uh common sense question answering or cqa data set we can see that star with rationalization performs on are with GPT 3's performance on directly fine-tuning on arithmetic so if we were to take gpt3 have a pre-trained model then fine-tune it on the just a question and answer for arithmetic it performs just as well as star with rationalization but on a Model that is literally 30 times smaller so we can see here that using rational generation and rationalization in Star genuinely does help in enhancing model per performance of a specific model size now let's talk about a couple of limitations of this approach so the first is that the base model should be able to or rather should be big enough to generate rationalizations or rationals and this is why exactly we use the gptj model when I was talking about the performance and not like gpt2 gpt2 is a smaller model of 1.5 billion parameters that doesn't really do well in rationale generation and so it doesn't really benefit from this star approach because it has to generate appropriate rationals if it needs to learn from it and the second issue is that star might generate cases where the answer is correct but the rationale might actually be wrong so for example in like the question answering case of multiple choice questions here is a question that was asked what might someone get from learning about science and these were the options that were presented with the correct answer being increased knowledge that is correct but the llm responded in a way where the answer was correct that it is the correct answer is increased knowledge but the rationale just was not really that correct and it gave the answer must be something that someone would get from learning about science learning about science would increase knowledge it's kind of an iffy rationale and rationale quality is also very difficult to assess because even if you crowdsource this generation or even evaluation of rationale different people have different metrics in how they think what is a good rationale and what is a bad rationale and so that also adds to to this issue here quiz time o this is going to be a fun one which of the following is an issue with star a star can generate wrong rationale with the right results and is difficult to track B Star cannot be used with models that can't generate good rationale C star can perform bad if less than 10 input prompts are used or d none of the above note that multiple answers here may be correct and I'll give you a few seconds to answer this question the correct answers are A and B but can you tell me why give your reasoning down in the comments below and let's have a discussion and if you think I do deserve it at this point please do consider giving this video a like because it will help me out a lot and that's going to do it for quiz time and pass three of this explanation but before we go let's generate a summary so in this video we actually talked about trying to understand why we would want some self-taught reasoning it's because there are some cases that in this language model it performs pretty well like in normal question answering but then there are some other cases like in arithmetic where it gets it wrong if you directly fine-tune now we can make use of Chain of Thought prompting in order to have the llm generate a rationale and hence also be better at eventually generating a correct answer so combining the ability of using rational to generate more accurate answers along with the ability to like finetune and also have the model learn on a specific task we can use star which involves generating a data set with rationale and then fine-tuning the llm on this data set we then talked about exactly how the star method can work in two of its processes which is rational generation as well as the rationalized step in order to leverage incorrect responses too we then talked about some performance of star compared to like GPT as well as its limitations and that's all we have for today if you like this video and do think I deserve it please do consider giving this video a like and if you are interested in more AI related content check this video out right over here thank you all so much for watching and I will see you in the next one bye-bye